Matrix balls, radial analysis of Berezin kernels, and 
hypergeometric determinants 

YuRii A. NeretinQ 



The subject of this paper is a natural one-parametric family of Hilbert spaces and repre- 
sentations interpolating on a Riemannian noncompact symmetric space G/K and on the 
dual Riemannian compact symmetric space Gcomp/K. The spaces Va consist of holomorphic func- 
tions on some symmetric space G/K D G/K, the limit of as a ^ +00 is L?{G/K), and the 
limit of Va as a tends to —00 taking integer values is L^{Gcomp/ K). Even the case in which 
G/K is the Lobachevskii plane U(l,l)/U(l) x U(l) and Gcomp/K is the two-dimensional sphere 
U(2)/U(l) X U(l), is nontrivial and is still not satisfactory clear today (see |57|). 

More formally, we consider the problem of the restriction of a unitary highest weight represen- 
tation of a semisimple group G to a symmetric subgroup (see 4.7 below). This formulation is short 
and simple but it does not explain why this restriction problem differs from a nondenumerable 
collection of other restriction problems. 

The objects of this paper first appeared in the short note of Berezin Q in 1978. Berezin perished 
two years later and a complete text of his work never was published. For the second time, these 
objects occurred in a joint work of G.I.Olshanskii and author, but it was published only partially 
in two short notes and []62[ . 

As a result, this problem was "lost" and it became visible only in 1994-95 in |7^ and [59| . 
Some other references in the last 5 years are Q, (these papers can 

also be the source for many other references). 

The present paper has three purposes. 

1) We intend to survey phenomena arising in the analysis of the Berezin kernels. We consider 
a special case 

G^Vip,q) 

In this case, there exist some specific tools that allow to avoid main difhculties existing in a general 
case. For instance, the complete Plancherel formula]^ can be proved in a very simple way^. 

2) For the case G = \J(p,q), we can also obtain some results that today are not achieved in 
the general situation. The main new result is an explicit construction of a unitary intertwining 
operator from L^(\J{p, q)/\J{q) x U(p)) to the Berezin deformation of L^. 

3) Many multivariate special functions appear in a natural way in the harmonic analysis of the 
Berezin kernels (in particular, multivariate continuous dual Hahn, dual Hahn, Meixner-Pollachek, 
Krawtchouk, Laguerre, Jacobi orthogonal polynomials. Jack polynomials. Jack zonal functions, 
matrix F-function and B-function, matrix Bessel functions, Heckman-Opdam multivariate hyper- 
geometric functions). Also, it seems to me that the analysis of the Berezin kernels leads to some 
"new" special functions (to the A- function, see Section 11, and to generalizations of Gross-Richards 
kernels, see section 10). The relatively simple picture for V{p,q) allows to touch some of these 
objects in a simple way. 

1 supported by the grants RFBR 98-01-00303 and NWO 047-008-009 

■^In the case of Hermitian symmetric spaces G/K the Plancherel formula for large values of the parameter a (see 
below) was announced in the Berezin work [fl, pr oof was published by Upmeier and Unterberp-er ip 1994 [Q; for 
rank 1 case it was obtained by van Diik and Hille , for general situation it was obtained in |55| . 



Partially this was done by Hille 129 
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It seems to me that the subject of this paper is elementary, this is some topic in analysis of the 
matrix variable. For this reason, I try to follow a simple approach to noncommutative harmonic 
analysis in the spirit of ||T^, [Q, js^ whenever 1 can. 

I am very grateful to Grigory Olshanski, Bent 0rsted and Vladimir Molchanov for meaning- 
ful discussion. I thanks the administration of the Erwin Schrodinger Institute for Mathematical 
Physics, where this text was prepared, for their hospitality. 
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1. Preliminaries: positive definite kernels 



Positive definite kernels machinery is a usual tool for work upon Hilbert spaces (|^, [p8[). 
This section contains simple general facts concerning this subject. 

1.1. Positive definite kernels. Let X be a set, let H he a Hilbert space, and let (•, •) denote 
the scalar product in H. Consider a map x t-^ Vx from X to H. We define the function L{x, y) on 
X xXhy 

L{x,y) = {vx,Vy) 



Obviously, L{x, y) ~ L{y, x) and for any xi, . 



G X the matrix 



/L{xi,Xi) ... L{xi,Xn)^ 



\L{Xn,Xi) ... L[Xn,Xn)/ 



il.l) 



is positive semidefiniteQ 

A function L{x, y) on the set X x X is called a positive definite kernel on X if L{x, y) = L{y, x) 
and for all xi, . . . , x„ € A the matrix ( |1.1| ) is positive semidcfinite. 

Nontrivial examples appear in Sections 3-4, 

Theorem 1.1. ifzol Let L{x,y) be a positive definite kernel on X. Then 

a) There exists a Hilbert space H and a system of vectors Vx & H enumerated by x £ X such 
that L(x,y) = {vx,Vy) and the linear span of the vectors Vx is dense in H. 

b) A space H is unique in the following sense. Let H' be another Hilbert space and let v'^ G H' 
be another system of vectors satisfying the same conditions. Then there exists a unique unitary 
operator U : H H' such that Uvx = v'^ for all x. 



'An n X n Hermitian matrix Q = {qij} is called positive semidefinite if ^ qijWiWj ^ for any 



wi, ■ 
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Proof. Let Vx, where a; e X, be formal symbols. Consider the linear space H consisting of all 
the formal finite linear combinations X]fc=i CkVxk: where Ck G C. We define a scalar product in H 

by 

n m n,Tn 

k=l 1=1 k=l,l = l 

The space if is a pre-Hilbert space and H is the Hilbert space associated with H. Kl 
We denote^ by 05 [L] = 05 [L; X] the space H equipped with the distinguished system of vectors 
Vx ■ The set of vectors G 03 [L] is called a supercomplete system, or an overfilled basis, or a system 
of coherent states. 

Remark. A supercomplete system is nothing but a system of vectors with explicitly known 
pairwise scalar products. Nevertheless the knowledge of these data can be an effective tool for 
workin in a given Hilbert space. □ 

If X is a separable metric space and L{x,y) is continuous on X x X, then the Hilbert space 
!B[Z/] is separable and the map x '—^ Vx is continuous. In all natural cases, these conditions are 
satisfied. For certain technical reasons, the general definition given above is more convenient. 

1.2. Functional realization 05°[L]. Consider the space 05[L] defined by a positive definite 
kernel L. To each h E ^[L] we assign a function fh{x) on X by 

fh{x) = {h,Vx)<slL] 

The linear span of the vectors Vx is dense in 03 [L] and hence the map h i-^- fh is an embedding of 
!B[L] to the space of functions on X. We denote by 03° [i] the image of this embedding. A scalar 
product in *B° [L] is defined by 

{fhJq)<s°lL] ■■= {h,q)<z[L] (1-2) 
The function 9a corresponding to an element Va of the supercomplete system is given by 

9a{x) = L{a, x) 

Proposition 1.2. (Reproducing property) For each / e <8°[L] 

(/,^a)<8=[L] -/(a) (1-3) 

Proof. 

{fh,Oa)<S'>lL] = {h,Va)<S[L] = 

1.3. Reconstruction of the kernel L from 03° [i]. Let 03° be some Hilbert space consisting 
of functions pl on X, and let the linear functional Ux{f) '■— f{x) be continuous on 03° for each x E X 
. Then (see [|67|, Theorem 2.4) for any x E X there exists a unique function 6x € 5B° such that 

{f,ex)^fix) 

We define a positive definite kernel L{x,y) on X by 

L{x,y) := {dx,9y)<s'' = Ox{y) = Oy{x) 

for all / e B° . Then the space 03° coincides with the space 03° [L] defined by the positive definite 
kernel L. 

The kernel L is called the reproducing kernel of the space 03° . 

^ in honor of S.Bergman, V.Bargmann and F.A.Berezin 
''The space L-^ does not consist of functions! 
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1.4. Convergence in 03° [L]. 

Lemma 1.3. (see, for instance, ^7\) Assume X is a complete metric space, and the kernel L 
is continuous. If a sequence fj G 05° [L] converges to f in OS°[L], then it converges uniformly on 
compact sets in X. 

Corollary 1.4. Any element o/!B°[L] can be approximated by finite sums '^CkL{ak,x) in 
the topology of uniform convergence on compact sets. 

1.5. Hilbert spaces of holomorphic functions. Let J7 be a bounded open domain in C". 
Assume a positive definite kernel u) on U, is antiholomorphic in z and holomorphic in u. Then, 
by Corollary 1.4, elements of the Hilbert space 05° [L] are holomorphic functions^ 

Let C,{z) be a continuous function on 17 and let C,{z) > for any z € fi. Denote by ^(il, C) the 
space of holomorphic functions on satisfying the condition 

/ \f{z)\\{z){dz}<^ 

where 

n n 

{dz} Yl diRezj) J| d{lmzj) 
denotes the Lebesgue measure on fl. Consider the scalar product 

if, 9)= I f{z)W)a^){dz} (1.4) 

JO 

in the space 55(57, C). 

Theorem 1.5 a) The space 'B{VL,C,) is closed in L'^{VL,C,)- 

h)For any u € the linear functional f i— s- f(u) is continuous on 3(57, Q. 

Proof, b) Let 51 be a polydisk \zj\ < rj, C,{z) — 1 and u — 0. Then /(O) coincides (up to a 
factor) with (/, 1). Thus in this case the statement is obvious. Consider an arbitrary point w G 51 
and a small polydisk il C 51 with center in w. Then 

/ |/(z)PC(z){dz}^minC(z) / \fiz)\Hdz} 

Hence the convergence /„ ^ / in S(51, C) implies the convergence /„ f in L^{D). Thus fn{w) 
converges to f{w). This implies b) 

The assertion of a) is a consequence of b) and Lemma 1.3. Kl 

The spaces 15(51, r) are called weight spaces of holomorphic functions. 

1.6. Another functional realization 05* [L]. Let X be a smooth manifold. Let a positive 
definite kernel L be C°°-smooth. Let £'{X) be the space of compactly supported distributions on 
X. We define a scalar product in £'{X) by 

(Xl,X2> = {i,Xl X2} 

where {•, •} denotes the pairing of smooth functions and distributions. We define the space 05* [L] 
as the Hilbert space associated with the pre-Hilbert space £'{X). 

The canonical unitary operator J : 05 [L] ?B*[L] is defined by the condition: Jv^ is the 
(5-function supported by x. 

Remark. In general, distributions do not represent all the elements of ?B*[i]. 

1.7. Some operations with positive definite kernels. 

^Recall the Weierstrass theorem. Let /„ be holomorphic and /„ converge to / uniformly on compact sets. Then 
partial derivatives fn converge to f uniformly on compact sets. In particular, / is holomorphic. 
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Proposition 1.6. Let Li, L2 he positive definite kernels on X. Then L1L2 and Li + L2 are 
positive definite kernels. 

Proof. Let Vx be the supercomplete system in and Wx be the supercomplete system in 

23 [L2]. Consider the space 53 [Li] (8) 53 [L2] with the supercomplete system Vx ® Wx and the space 
23[Li] ® 5B[L2] with the supercomplete system Vx © Wx- Both statements are now obvious. Kl 

Proposition 1.7. Let L{x,y) be a positive definite kernel on X . Then for an arbitrary function 
7(x) on X 

a) the kernel M(x,y) :— L{x,y)^{x)j{y) is positive definite. 

b) the operator 53° [i] 53° [M] given by f{x) i—f f(x)"f{x) is unitary. 

Proof. Consider the vectors j{x)vx G 53[i]. M 
2. Groups U{p, q) and matrix balls 

Assume that p ^ q. 

2.1. Groups U(p, q). Consider the space C^QC^ equipped with the indefinite Hermitian form 
Q defined by the {p + q) x (p + q) matrix ( J ."^ ) . We denote by e^, . . . , 6+ , ej", . . . e~ the standard 
basis in C ® C^. The pseudounitary grouj^ G = U(p, 5) is the group of all linear operators in 
C © preserving the form Q. In other words, the group U(p, q) consists of all (p + (?) x (p + q) 
matrices satisfying the condition 



a b 
c d 



1 

-1 



1 

-1 



(2.1) 



2.2. Another realization of \J{q,q). Assume p = q. Consider the new basis 

7f(e^ +ej-),...,..^ — 75V 



V2 



"q ■ 



-^(e+ 



(2.2) 
(2.3) 



in ® C*. The matrix of the Hermitian form Q in this basis is ( 1 J ) • Thus the group \J{q, q) can 
be represented as the group of all {q + q) x {q + q) matrices {c^)) satisfying 



A B 
C D 



A B 
C D 



1 

1 



(2.4) 



2.3. Another realization of U(p, q). Let us realize the groups U(p, q) and U(g, q) as in 2.1. 
Consider the natural embedding of U(p, q) to U(g, q) given by 



h I 



Ig-p 




where Iq-p denotes the unit matrix of the size q ~ p. Consider the realization of U(<;, q) described 
in 2.2. Then U(p, q) becomes the group of {{q — p) + p + {q — p) + p) x {{q — p) + p + {q — p) + p) 
block matrices g satisfying the conditions 










1 











1 






(v\ 

















1 


9* = 











1 












.9 


1 











1 











; 9 


V 




V 






1 





0^ 






1 





0^ 




\v 







(2.5) 



for all u e C P. Sometimes this model is useful. 

2.4. The symmetric space G/K = \J{p, q)/\J{p) x \J{q). We say that a p-dimensional 
subspace i? C © is positive, if the Hermitian form Q is positive definite on R. Denote by 



'We also denote by U(n) the group of unitary n X n matrices. 
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Gup g the space of ah positive p-dimensional subspaces in C?. The group U(p, g) acts on 
® C and hence it acts on Gr^ g. Obviously (by the Witt theorem), this action is transitive. 
The stabilizer of the subspace G Gfp g consists of matrices having the form 



e U(p), b e U(g) (2.6) 



a 
d 

Hence the space Gr^ is the homogeneous space 

G/K = U(p,g)/U(p)xU(g) 

In this paper we fix the notation 

G = U(p,g); K = U(p)xU(g) 

2.5. Cartan matrix balls. Assume p ^ q. A matrix ball Bp.g is the space of all complex 
p X q matrices with norm^ less than 1. Also, z G Bp^^ iff zz* < 1. 

For z S Bp^g we define the subspace Graph{z) C C consisting of all the vectors of the 
form {v,vz), where v g C^. It can easily be checked that the map z i-^ Graph{z) is a bijection 
Bp,g Gip Thus the space Grp ^ ~ G/K is parametrized by points of the matrix ball Bp g. The 
action of the group G in these coordinates is given by the linear-fractional transformations 



z ^ zJs] (^a + zc)-\b + zd) (2.7) 



where g = e G. 



The complex Jacobian of the transformation (2.7) is 



det(a + zc)-P-« det (2.8) 

Lemma 2.1. The G-invariant measure on Bp,g is given by 

det(l - zz*yP-''{dz} (2.9) 

where {dz} is the Lebesgue measure on Bp ^. 

Proof. This follows from ( |2.8| ) and the simple identity 

1 - zlsl (ulfl)* = (a + zc)-\l - zu*){a* + c*u*)-'^ (2.10) 

2.6. Structure of the boundary of the matrix ball. Denote by Bp g the closure of Bp g in 
CP'', i.e the set of matrices with norm ^ 1. Denote by Mk the set of all matrices z £ Bp g satisfying 
the condition 

rk(l - zz*) = A;; k = 0,...,p-l 

The following statement is trivial 

Lemma 2.2. The sets Mk are exactly the orbits of\]{p,q) on the boundary ofUp q. 
Remark. The orbit Mq is the Shilov boundary of Bp^g, see ||l^, X.5. □ 

2.7. Siegel realizations of G/K: matrix wedges. The Cartan realization of G/K will be 
basic for us. Nevertheless in some places we shall need of the Siegel realizations. PI 

^The term norm in this paper means the norm of an operator from Euclidean space to Euclidean space C. 
The notation A > means that the operator A is positive i.e., {Ax, x) > for all x 0. The notation A > B means 
A- B>0. 

^"The action of the compact subgroup K C G has the simplest form in the Cartan realization; if we want to 
write formulas related to the parabolic subgroup (see Subsection 6.2 below), then the Siegel realizations are more 
convenient. 
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Assume p = q. Denote by Wg the space of all q x g complex matrices T satisfying the condition 



T + T* >0 



Consider the basis (2.2)-( p.3[ ), denote by Z+ the subspace in C ® C spanned by wj' and 
denote by Z_ the subspace spanned by wJ . For T S Wg consider the operator T : Z+ Z_. It 
can easily be checked that the map T t-^ Graph{T) is a bijection Wg ^^tg 



The action of the group G = V{q, q) on the wedge Wg is given by 

T Tifl := {A + TCy^{B + TD) 



(2.11) 



where the matrix g — (c' B) satisfies (2.4). 

The spaces Bg^g and Wg are identified by the Cayley transform 



T : 



-l + (l + z)-i 



2.8. Siegel realization in the case q ^ p: sections of wedges. We define the space SWp g 
as the subset in Wg consisting of all block {{q — p) + p) x {{q — p) + p)-matrices of the form 



S 



2K L^Wg 



(2.12) 



The group U(p , q) acts on the space SWp.g by the same formula (2.11) for a matrix g satisfying 
equations ( |2.5| ), for details see Q. 

2.9. Radial part of the Lebesgue measure. The subgroup K — U(p) x lJ{q) C G acts on 
Bp.g by the transformations z i— s- a^^zd (see (2.6), (2.7)). Obviously, any element z € Bp_g can be 
reduced by such transformations to the form 



/Ai 
A2 

Vo 



... 0\ 
... 



A„ 



0/ 



where 



1 > Ai ^ A2 ^ 



^ Ap ^ 



(2.13) 



are the eigenvalues of y/zz*. We denote the simplex ( 2.13| ) by Ap. 

Consider the map tt : Bp^g — > Ap taking each z g Bp_g to the collection of the eigenvalues of 
\J zz* . The image of the Lebesgue measure on Bp ^ with respect to the map tt is the measure on 
Ap given by 

const. n(A?-A,^fn^'^'"'^^'n'^^. 

(see X.l), see also numerous calculations of this type in Chapter 3). 
Here and below the symbol 'const ' denotes a constant depending only on p,q. 



Thus the image of the G-invariant measure (2.9) is 

p 



const- n (A?-A2)2[]a; 



2(g-p) + l 



It will be convenient for us to define new coordinates 



ii-x^^)-p-^l[dx, 



(2.14) 



A?/(l-A?) 



(2.15) 
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Then the simplex Ap transforms to the simphcial cone Xp 

Xp : xi ^ X2 ^ ■ ■ ■ ^ Xp ^ (2-16) 



The measure (2.14) in the coordinates xj is of the form 

n (xk-xiffidxk (2.17) 

j = l l^fc<i<J9 j = l 

Below we consider K.-invariant functions on G/K as symmetric functions in the variables 
Xj ^ or as functions on the simplicial cone X. 

Theorem 2.3. (Hua) 

det(i - ..*)— = - n 

^3 fc=l ^ ' 

where uj is the volume of Bp^q . 

The problem is reduced to integrating of the function J|(l + Xk)~" over the measure ( ^.17 ). 
This is a special case of the Selberg integral, sec [Q, chapter 8. Hua's calculations (0], chapter 
2) are interesting and important by themself. For other ways of calculation see ||l^, [^4[ . 
2.10. Comments. A matrix hall (see |^^) B is 

— the set of all p x g matrices over R, C or the quaternion algebra H such that zz* < 1 

— or the set of all n x n matrices over R, C, H satisfying zz* < 1 and the natural symmetry condition. 
The natural conditions of symmetry are 

2 = 2* and z = —2* over R; 

2 = 2*, z = —2*, and z = z* over C; 

2 = 2* and 2 — —z* over H. 

We consider the group G of linear-fractional transformations (2.7) preserving B. Denote by K the 
stabilizer of the point G B. All classical Riemannian noncompact symmetric spaces G/K can be obtained 
in this way. The table is contained in Appendix A, see also jHj]. 

Tools that are used below cannot be applied to arbitrary matrix ball. 

3. Spaces of holomorphic functions in matrix balls. Berezin scale 

A subject of this section is standard. Usually we give sketches of proofs. 

3.1. Berezin scale: large values of parameter. Let a > p + q — 1. Consider the weight 
space 

i?„ := S(Bp,q,det(l - zz*)°'-P-'^{dz}) 
consisting of holomorphic functions on Bp ,j (see 1.5). The scalar product in Ha is defined by 



(/, g)a = Cia)-' / fiz)g{z) det(l - zzT^^-^dz} (3.1) 



We define the normalization constant C{a) by (2.18), then (1, 1)q, = 1. 

Let us define the operators r^ig), where g = ( ^ ^) e G = U(p, q), in Ha by 

Ta{g)f{z) = f{{a + zc)-\b + zd)) det(a + zc)"" (3.2) 

A simple calculation (with applying ( |2.8| ), ( 2.10| )) shows that the operators Ta{g) are unitary in 
Ha- Hence we obtain a unitary representation of G = U(p, q) in Ha- 

Remark. If a is integer, then the expression det(a + zc)"" is well defined. Otherwise the 
function 

det(a + zc)-"' = det(a)" det [(1 + zca-^)""] = e"(''^'^°' a+2nki) ^(^^ ^ zca^i)^"] 
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on Bp^q has an infinite number of holomorphic branches, which differ by a constant factor. Indeed, 
imphes ||a|| > ||c||, hence ||ca^^|| < 1, hence the expression [. . .] is well defined. Thus, for a 
noninteger a, the operators Ta{g) are defined up to a scalar factor e^'^*"'^, and the representation 
Ta is projective (see |Q, 14). We can also consider the representation Ta as a representation of 
the universal covering group of the group U(p, g), see [|4|, Corollary to Theorem 14.3.1. □ 
Theorem 3.1.(Berezin, The reproducing kernel of the space Ha is 

Proof. We must find functions 9a G Ha such that {f,Oa) — f{a)- Expanding f{z) into its 
Taylor series, we obtain 

^ ^ f{z)Aet{l~zz*r-P-'i{dz}^-^ [ {f{Q) + ...)Aet{l-zz*r-P-'^{dz} = 



C{a)J^^/' ' ' ' ' ' Cia) 



f{0) det{l ~ zzT-'-^dz} = f{0) 



Thus 9o{z) = 1. Let us evaluate {Ta{g)f,Oa)a in two ways. First, it equals 
{f{zM)det{a + zcy, eo)a = f{a^^b) det(a)-" 
The operators Ta{g) are unitary, hence it is equal to 

{f,Ta{g-^)e^)a = {f, {a-b*z)-")a 

and we obtain an explicit expression for Kl 

Formula (3.1) defining the scalar product in Ha has sense ioi a > p + q — 1. Nevertheless we 
shall see (Theorem 3.8) that the reproducing kernel Ka is positive definite for a > p — 1 and also 
for a = 0, 1, . . . ,p — 1. Subsections 3.2-3.3 contains the preparation to the statement and proof of 
Theorem 3.8. 

3.2. Preliminaries: K-harmonics in the space of polynomials. Let 

Ml ^ ^ ■ • • ^ Mri ^ 

be integers. The Schur function is defined by 

det[yf+"--'] det[yr^+"-^] 



SA*(yi,---,2/n) = s"(yi,...,?/„) 



det[y1 ^] Y{k<iiyk~yi) 



where yi € C (see 1.3; Q, §73). The numerator is zero on the hyperplanes yj — yk and hence 
is a symmetric polynomial in yi, . . . , ?;„. It is easy to prove that 



s 



n+m 
All,... 



A.„,0,...,o(2/l' • ■ • , yn, 0, . . . , 0) = S^i,. (2/1, ■ • ■ , yn) (3.3) 



Let Ahe a n X n matrix. Let yi, . . . , y„ be the eigenvalues of A. We define the Schur function 

SM) by 

S^{A) = Sf,{yi, . . . ,y„) 

By Pp, = p^-^ ^„ we denote the representation of GL(n,C) with the signature p (see ||8^, §49; 
[ ^ ; these objects are used only in this Subsection). The Schur function S'p(A) is the character 
of p^, i.e., <5'^(^) = tr/3p(A). Recall (see ||8^, §73; see also Theorem 1.2.4 on eliminating of 
undeterminacy) that the dimension of the representation p^ is given by the Weyl formula 

dimp;: = .,(l,...,l)= n (3.4) 
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Denote by Matp,, the space of all complex p x q matrices. Denote by Polp,q the space of all 



holomorphic polynomials on Matp,q. The linear-fractional transformations (2.7) for g = (g^) GK 
reduce to the form 

T^ig)fiz) = f{a-'zd)detia)-" 
Let us omit the unessential scalar factor det(a)^" and consider the action of \J{p) x U(g) given by 

fj : f{z) ^ .f{a-'zd) = f{a*zd) 

It is natural to extend this action to a GL(p, C) x GL((7, C)-action given by the formula f{z) i— s- 
Theorem 3.2. (Hua, ||, see also [|o|, §56) 

PoW^ P^„...,,,®P^,,...,,„o,...,o (3-5) 
This theorem is a consequence of the following Lemma. 

Lemma 3.3. Denote by Aj the determinant of the left upper j x j block of z. All GL(p, C) x 
GL((7, C)-highest vectors in Polp^^ have the form Y[^=i '^j^ ^^^^ • 

Proof of the Lemma. Denote by Nq the group of aU d e GL(q, C) with units on the 
diagonal and zeros under the diagonal. Denote by the group of all a e GL(p, C) with units on 
the diagonal and zeros over the diagonal. Denote by Ap, Aq the diagonal subgroups in GL(p, C), 
GL(g, C). We want to find all eigenfunctions of the subgroup ApN'^, x AqNq C GL(p, C) x GL(q, C). 
Denote by the set of all w e Matp^^ such that Wij = for i ^ j 

A highest vector is a iVp x iVg-invariant function on Matp For a matrix z € Matp^q in a general 
position there exists H G that can be represented in the form H = TzS, where T G N^, S G Nq 
(it is sufRcient to apply the Gauss elimination algorithm). Moreover, the diagonal matrix elements 
hjj of H are given by hjj — Aj/Aj_i. Hence a highest vector is a rational function depending on 
A,/A,_i, j = l,2,...,p. 

Now we call to mind that a highest vector is an Ap x Ag-eigenvector. Kl 

Denote by Pol^^ the subspaces in Polp^^ corresponding to the summands in ( |3.5| ). 

Lemma 3.4. a) The function S'^{uz*) is a positive definite kernel on Polp q. 

b) S^P^_^^{uz*) = Sl_^^^,_,{z*u) 

c) The subspace Polp^ coincides with the space *B° Matp.g] defined by the positive definite 
kernel S^{uz*). 

Proof, a) Obviously, the representation extends canonically to the representation of the 
semigroup T of all operators C^, and p^(A*) — p^(A)* (see, for instance [|9|, Chapter I, 

Appendix A, 3.3). 

Let us show that ti p^{AB*) is a positive definite kernel on P. Let Vj be an orthonormal basis 
in pP. Then 

The summands are positive definite (since Vj ^ Pf^{A*)vj is a function from P to Euclidean space, 
see 1.1) By Proposition 1.6, the sum also is positive definite. 

Now we embed Matp^q to P adding [q — p) zero rows at the bottom of the matrix. 

b) The nonzero eigenvalues of uz* and u* z coinci de (l et /i be a vector; then zu*h = Xh implies 
{u*z)u*h = \u*h). Thus the statement follows from ( ^!3|) . 

^^The groups U(p) X U(q) and GL(p, C) X GL(q, C) have the same invariant subspaces in Polp q. Indeed a subspace 
of homogeneous polynomials of a given degree is invariant with respect to both groups. It remains to apply the 
Weyl's unitary trick, see po| |, §42. 
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c) First, the kernel Sfj,{uz*) is K-invariant and hence it defines a K-invariant subspace. 

The elements of Polp,q are holomorphic functions and hence they are uniquely determined 
by their restrictions to the submanifold Mq defined in 2.6. The elements of Mq are isometric 
embeddings ^ C. Consider the natural map U(g) Mq: we take a unitary matrix and delete 
its last q — p rows. Hence a functions on Mq can be regarded as a function on U(g). 

The puUback of the kernel S^{z*u) from Mq to \J{q) is 









(.,.( 





2) 



The kernel £^(31,52) 5'^(gi.92 ^) on U(g) defines the space p^,,...,^p,o,...,o P^n,...,A<p,o,...,o- % 
Corollary 1.4, the space 7J°[L^] is the linear span of the functions Sf^{gA), where A ranges over 
invertible matrices. This space is finite-dimensional, and thus it is closed with respect to point- 
wise convergence. Hence it contains functions Sfj,{gA) for all noninvertible matrices A. Thus 
^°[K^;V{q)] C H°[L^,\J{q)]. Hence the representation of U(g) in 05° [5^; Matp,g] is the direct 
sum of several copies of P/ii,...,pp,o,...,o- It remains to apply Theorem 3.2. 

3.3. Expansion of the Berezin kernel Ka- 

Theorem 3.5 (Hua H) 

p 

Y[i^-yjr"^ c{fii,...,fip;a)Sf,,,...^f,^{yi,...,yp) 

j=l fj.i^...^fip^O 

where 



= d.mp5;^,...,,^ n r(a-, + l)(M.+p-,)! 



(3.6) 



Proof is contained in |32 , Theorem 1.2.5, |15[ . 
Corollary 3.6. (Berezin 

det(l — uz*)^" = ^ c(^i,. 



(3.7) 



define the same scalar product 



where the constants c{fj,;a) are the same as above (3.6). 

Corollary 3.7. The kernels c{fj,; a) Sfj_{uz*) and dct{l — uz* 
in the subspace Pol^^. 

Proof. The if^-scalar product in Pol^ ^ is U(p) x U((7)-invariant. The scalar product in Pol^ ^ 
defined by S^{uz*) also is U(p) x U(g)-invariant. The U(p) x U(g)-module Polp,j is irreducible. 
Hence these scalar products differs by a scalar factor. Denote by a^S^{uz*) the kernel defining 
the iJa-scalar product in Polp The subspaces Polp ^ are pairwise orthogonal, and hence (see the 
proof of Proposition 1.6) we have det(l — uz*)~°' = j2 ""/j 5*/^ (wz* ) . Kl 

3.4. Berezin scale: general values of parameter a. 

Theorem 3.8. (Berezin [g, and also ||l^, Q) a) The function Ka{z,u) = det(l-uz*)-" 
is a positive definite kernel on Bp^g if and only if a belongs to the set 



a = 0, 1,2, ... ,p- 1, 



a > p — 1 



(3.8) 



b) Denote by Ha the Hilbert space of holomo rph ic functions on Bp^g defined by the kernel Ka 
Then the operators Ta{g), g G U(p, g), given by (3.2) are unitary in Ha- 



ll 



c) If a > p— 1, then the space Ha contains all holomorphic polynomials. If a = fc = 0, 1, . . . ,p — 
1, then Ha has the following V{p) x \J{q) -module structure 

— ^ P^ni,...,tik,0,...,Q'^ Pjii,....fJ.k,0,...fi 

Proof. A sum of positive definite kernels is positive definite (see Proposition 1.6). Hence it is 
sufficient to check positivity of the coefficients in the expansion ( |3.7|) . The last item is trivial. Kl 
Remark. The space Hq is one-dimensional, it contains only constants. 

Remark. The space Hq coincides with the Hardy space i?^(Bp.g). The scalar product in Hq 
is given by 



(/, 5)9 = const / f{z)g{z){dz} 

J Mo 

where Mq is the boundary orbit defined in 2.6 and {dz} is the unique K-invariant measure on Mq. 

3.5. Berezin scale: small values of the parameter a. Consider the case a — k — 

0, 1, . . . ,p — 1. Let 

Okl 



dzki 



Consider the matrix 



(dii ... dlq\ 
: ■.. : (3.9) 
dpi . . . dpqj 

We regard the minors of the matrix D as differential operators. 

Theorem 3.9. Let f e Hk, k = 0, 1, ... ,p — 1. Then each (fc + 1) x (fc + 1) minor of D 
annihilates f , i.e., for all ii < 12 < ■ ■ ■ < ik, Ji < ^2 < • ■ ■ < jk the following identity holds 



det 



f{z) = (3.10) 



Proof. By Corollary 1.4 and holomorphness of /, it is sufficient to prove the statement for 
elements of the supercomplete system, i.e., for the functions 

f{z) = dct(l - za*y'' 

Obviously, the system of partial differential equations ( [j.lOj ) is invariant with respect to the trans- 
formations 

z ^ AzD, where A e GL(p, C), D e GL(g, C) 
Hence it is sufficient to consider the case in which block p x {p + {q — p)) matrix a has the form 

a = (Ip 0) 

where Ip is the unit p x p matrix. Now f{z) = det~'^(l — za*) depends only on left p x p block of 
z. Hence, without loss of generality, we can consider only the case p = q and f{z) = det(l — z)^^. 
But the system (3.10) is invariant with respect to translations and we can change our function 
f{z) to det(z)-'=. 

A direct (but pleasant) calculation shows that 

{dnd22 - 921921) det(z)-i = 
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This implies that 3x3 minors of D annihilate det z ^ = det z ^ det z ^ etc. Kl 
Remark. Let / be a solution of the system (3.1C) in Bp^g. Generally, / ^ i/fe, since / 



can have too rapid growth near boundary. Nevertheless / can be approximated by finite sums 
det(l — za*^~^ in the topology of uniform convergence on compact sets. □ 

3.6. Gindikin—Vergne— Rossi description for Ha- We consider only the case p = q. First 
we observe that the Berezin kernel in the wedge realization has the form 

K^{T,S)^dei{T* + Sy (3.11) 

We denote by Ha{Wq) the space of holomorphic functions on Wq defined by the kernel (3.11). 
The group 11(5,(7) acts in Ha{Wq) by the transformations 

/(T) ^ f{A + TC)-\B + TD)) dct{A + TC)"" 

where the matrix £) satisfies condition (2.4). 

Remark. Let Z e Wg. Let \j be the eigenvalues of Z, and Vj be the eigenvectors. Then 

0<{{Z + Z*)vj,Vj) = {Zvj.Vj) + {vj^Zvj) = {\j +\j){v.j,v.j) 

Henc e Re \j > and hence the function det Z^" :— Xj" is well defined on Wq. Thus expression 
( ^.11 ) makes sense. 



Consider the cone PoSg consisting of all positive semidefinite complex q x q Hermitian matrices 
X. The group GL{q, C) acts on PoSg by the transformations h : X i~> h*Xh, h E GL{q, C). Denote 
by Nk, where k — 0,1, . . . ,q, the set of all matrices X S PoSg such that rkX = k. Obviously, the 
sets Nk are the GL((7, C)-orbits on PoSg. 

Let x{^) be a tempered distribution supported by PoSp. Its Laplace transform is 

X{T)= ( exp(-trXT)x(X)dX (3.12) 

J PoSq 

The function x(^) is a holomorphic function of polynomial growth on the Siegel wedge (see 



Theorem 3.10. ( |19| , |73| ) Let a satisfy the positive definiteness conditions (3^). Then 
det(T)~" is the Laplace transform of some positive measure Va on PoSg. If ol > q — 1, then 
Va = detX"~'dX. For a = fc = 0, 1, . . . , 5 — 1, the measure Uk is the unique up to a scalar factor 
Slj{q,C) -invariant measure on the boundary orbit Nk- 

Proof, a) Let a > q — 1. We must check the equality 

/ det X"-«exp(-tr XT) dX = const- det (T)-" (3.13) 

POSg 

Since the both parts are holomorphic in T e Wq , it is sufficient to consider the case T = T* . Then 
the integral converts to 

f det X"-« exp(- tr T^/^XT^^^) dX = 

J POSg 

= det r-" j det(ri/2xri/2)"-« exp(- tr T^/^XT^''^) d{T^^^ XT^/'^) = 

= detT-"-/ dety"-«exp(-trr)dr (3.14) 

as required. 
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In fact, we only use the GL(g, C)-homogeneity of the measure Va'. 

Vo,{h*Ah) = I det(/i)|2"-2v„(A) (3.15) 

where h G GL(g, C), and A is a subset in Pos,. 

b) Let a ^ q — I. First we must check the existence of a SL((7, C)-invariant volume form on 
TVfc. We have Nk = SL(gf,C)/5£, where 3{ is the stabilizer IK of the point Uk ( o*" o) ^ ^fc- The 
subgroup 3i consists of matrices 

^ (o 5) ' ^ ^ ^ ~ ^' ^'^^ det(i?) det(5) = 1 

The group !K has no homomorphisms to M. Hence the (unique) volume form on the tangent space 
to Nk at the point Uk is iJ-invariant. Then we define a volume form on Nk by SL(n, C)-invariance. 

Secondly we must evaluate the Laplace transform of the me asure Vk- It is not hard to check, 
that the measure i^k satisfies the same homogeneity condition ( 3.15 ), a ^ k, and this gives the 
formula for the Laplace transform of z^fc. M 

For the case p ^ q, this gives an independent proof of Theorem 3.8 about positive definiteness 
conditions, i.e., we obtain the following corollary. 

Corollary 3.11. The kernel Ka{T, S) = det(T*+5)^" on Wg is positive definite if a > q—1 
or a = 0,1, ... ,q — 1. 

Proof. We use directly the definition of positive definiteness. We must show positivity of the 
expression 



det(r,!+rO-"ae/=^a-e; / exp(- tr(rfc + T^X) di.„(X) = 

= / I V Cfc exp(- tr TfcX)|2dzy„(X) ^ 

as required. Of course, we repeated the proof of the trivial part of the Bochner theorem (see [ |67| , 
V.2, Theorem IX.9). Kl 
Theorem 3.12. The Laplace transforn^ 

f{T)^ I f{X)eM-XYXT)dv^{X) (3.16) 

JPos, 

is a unitary (up to a scalar factor) operator from L^{Y'oSq,Va) to the Berezin space Ha[Wq]. 

Proof. Consider the family of functions >ca{X) = cxp(— tr A*X), where A e Wq, on Pos^. It 
is sufficient to prove that 

(><A,><i3)L2(Pos,,i.„) = const • {}iA,>CB)H^[Wg) (3.17) 

By ( 3.13| ), the left-hand side is const • det(^* + _B)~". By the same formula ( 3.13| ), ka{T) — 
det(A* +T)~°'. These 3<a are the elements of the supercomplete system of Ha{Wq) and we know 
their scalar products. Kl 

Remark. Assign xa be a supercomplete system in i^(PoSg, i^a)- Then the Laplace transform 
( p.l6| ) coincides with the transform *B — > 5B° described in 1.2. □ 

Remark. Theorem 3.10 implies Theorem 3.9 in the case p = q. For definiteness, assume 
a — q ~ I. Let D be given by (3.9). A (5-distribution f{X)vq-i satisfies the condition detX • 

f{X)vq^i = 0. After the Laplace transform, we obtain D f{X)vq^i = (this is not a complete 
proof). □ 

^■^We slightly change the notation with respect to (5.12) 
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3.7. Comments. 1) The construction described in this Section for the symmetric spaces U(p, q) /U(p) x 
U(g) works for all Hermitian noncompact symmetric spaces, i.e., also for Sp(2n, K)/U(n), SO*(2n)/U(n), 
SO(p,2)/SO(p) X SO(2) and two exceptional spaces, see §, Q, Q. 



2) The constant in formula (3.13) is a special case of Gindikin's matrix F-function [g^, see also the 
exposition in ||l5| . 

3) After Theorem 3.12 there arises a problem of transfering of the action of the group U(g,g) to 
the space L^(PoSg, i^a). The Lie algebra u{q,q) acts in L^{PoSq,Va) by second order partial differential 
operators, which can easily be written explicitly. The exponents of these differential operators (i.e elements 
of the group \J{q,q) itself) are integral operators with kernels involving matrix Bessel functions, see [ psf , 

0], ill- 

Consider the case G — U(l, 1). Then Wq = Wi is the Lobachevskii plane ReT > 0. The transformation 
T I— > of Wi corresponds to the Hankel transform in L^(R+, Va) (the Tricomi theorem). 

Rotations of the Lobachevskii plane with center at T = 1 corresponds to the Kepinski and Myller- 
Lebedeff ^] explicit solution of the Cauchy problem for the Schrodinger equation 

-F(x,t) = (-^+. +-,)F(.,t) 



4. Kernel representations and spaces 
4.1. Definition of kernel representations. Let 

a = 0, 1, 2, . . . ,p — 1, or a>p— 1 

Consider the kernel La{z, u) on Bp^g given by 

Laiz,u) :=|det(l - zu*)\-^°' = | det(l - == (4.1) 



= det(l - zu*)-"det(l - zu*) 

The kernel La is the product of two positive definite kernels and, by Proposition 1.6, La is positive 
definite. We denote by Va the Hilbert space *B°[Lq] defined by the kernel La- 
The group G = U(p, q) acts in the space Va by the unitary operators 

Ta(l j)/(z)=/((a + zc)-i(6 + ^d))|det(a + zc)r2" (4.2) 

We say that the representation Ta is a kernel-representation of the group G = U(p, q). 

4.2. Decomposition of the kernel rep rese ntations into a tensor product. Consider 
the unitary representation Tq of G defined by (3.2) and the complex conjugate representation Tq. 
Consider the tensor product Tq. The space Va := Ha ® Ha of the tensor product is the space 
of holomorphic functions on Bp g x Bp^^ defined by the positive definite kernel 

L(zi, Z2; Ml, 1*2) — det(l — uizj)^" dct(l — ^2^2)^" (4.3) 
and the group G acts in Va by 

F(zi,Z2) i-> 

F[{a + zicy^(b + zid), {a + Z2c)"^(6 + Z2d)] det(a + zic)"" det(a + Z2c)"" (4.4) 

Denote by A the diagonal zi = 'Z2 in Bp.g x Bp^g. 

First a holomorphic function on Bp.g x Bp ^ is determined by its restriction to A. Hence we 
can consider the space Ha ® Ha as a space of functions on A. The reproducing kernel of the latter 
space can be obtained by the substitution zi = 'Z2, ui = U2 to ( |4.3| ), see 1.3. 



15 



Secondly the transformations in square brackets in formula (4.4) preserve A. Let us rewrite 
the operators (4.4) as operators in the space of functions on A. This gives (4.2). Thus we obtain 
the following proposition. 

Proposition 4.1. The operator of restriction of functions F{zi,Z2) to A is a unitary G- 
intertwining operator from to the kernel representation T^. 

4.3. Representation in the space of Hilbert Schmidt operators. Let Wi, W2 be 

Hilbert spaces. Let A be an operator A : W\ Wi. Let be the matrix elements of A in 
some orthonormal bases. Recall that A is called a Hilbert- Schmidt operator (see v.l, VI.6) if 
J2 loijP < 00. 

The space IK§(Wi,W2) of all Hilbert-Schmidt operators Wi —> W2 is a Hilbert space with 
respect to the scalar product 

{A,B)^s ^tvAB* (4.5) 
There is an obvious canonical identification 

Hence we can identify 

Va — Ha ^ Ha — '}{§{Ha, H^) 

The group G acts in JC§{Ha, Ha) by the conjugations 

g:A^Ta{gr'ATaig) (4.6) 

4.4. Another version of the kernel L. Let us define the modified Berezin kernel 

^ , , dct(l - zz*)"det(l - uu*)" , , 

La{z,u):= ' , / ^TT^ — (4.7 

^ ' ' |det(l - zu*)|2" ^ ' 

We denote by Va the space *8°[iLQ.]. The kernel La is G-invariant in the following sense 

La {{a + zcy^{b + zd),{a + ucy^{h + ud)) ^ Za{z,u) (4.8) 



(this easily follows from ( 2.10 )). Hence the operators 

^"(c 'i)fi^) = f{i'^ + ^^r'ib + zd)) (4.9) 

are unitary in Vq. 

Remark. The last formula does not depend on a. Nevertheless we shall see (Section 7) that 
generally the unitary representations Tq, are nonequivalent. □ 

We also define the distinguished vector S = Sq G Vq by 

Eaiz) := det(l - zz*)-°' = Za{z, 0) (4.10) 

The vector Sq, is an element of the supercomplete basis and the G-orbit of Sq consists of all 
elements of the supercomplete system. Hence the vector Sq, is cyclicp^. 

By Proposition 1.7, the kernels L and L differ unessentially, and the canonical operator li : 
Va — *■ Vq,, described in Proposition 1.7, is given by 

U/(z) = /(z)det(l-zz*)" 



^''Let p be a representation of a group G in a topological vector space W (see | |34| , 7.2). A vector w ^ W is called 
cyclic if the linear span of the vectors p{g)w, g S G, is dense in W. For a subset R r W we define its cyclic span 
as the closure of the linear span of all vectors p{g)w, where g £ G and w £ B, see |34j, 4.4. 
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This operator intertwines (4.2) and ( [4.9D . Thus we identify the unitary representation Tq. and Ta 
of the group G. 

The last description of the kernel representation (kernel La and the representation Tq) will be 
main for us below. 

4.5. Limit of kernel representations as a ^ oo. Consider the space *B*[iLQ,] defined in 
1.6. Consider elements of Q3*[£;q,] of the form 

^{z)dct{l- zz*)-P-'^{dz} 

where (p{z) are compactly supported smooth functions on Bp q. The Q5*-scalar product of such 
distributions is given by 



C(a)-i // i:c{z,u)ip{z)^{u)detil- zz*)-P"'det{l-uu*yP~'^{dz}{du} (4.11) 



We define the normalization constant C(a) as the right-hand side of ( 2.18 ). By (O), this scalar 
product is G-invariant. 

Lemma 4.2. The family of distributions 

C{a)-^iiaiz,u) det(l - zz*)-p-« det(l - uu*)-P-'i{dz}{du} 
converges to det(l — uu*)~P~'^d{z — u) as a +oo. 



Proof. By the invariance property (4^), it is sufficient to follow only limit behavior of the 
family of distributions 

f2a{z) C{ay^L{z, 0) det(l - zz*)-P-i{dz} C{a)-^ det(l - zz*)°'-P-'^{dz} 

By the choice of the normalization constant, the integral of the function f2a is 1. We also have 
det(l — zz*) = 1 if z = and det(l — zz*) < 1 otherwise. The last two remarks impliy the 
convergence f2a{z) — > d{z) as a +oo. Kl 
Lemma 4.2 implies the following statement. 



Theorem 4.3. The family of scalar products (4.11) tends to the following L^-scalar product 



with respect to the G-invariant measure det(l — zz*) p ^{dz}; 



((p,V)= / (p{z)':P{z) det{l - zz*)-P-i{dz} 



We observed that the natural limit of kernel representations Tq as a — > +oo is the space on 
Riemannian symmetric space \](j),q) /\5{p) x \5{q). 

4.6. A canonical basis in space of K-invariant functions. Denote by the space of 
all functions f E Va that are invariant with respect to the group K — \J{p) x \J{q). 

Proposition 4.4. Let fi be a collection of integers ranging the domain ni ^ . . . ^ fip ^ if 
a > p — 1, and /xi ^ . . . > /Xfc ^ Hk+i = -- -=pLp = Oifa = k^p— 1. Then the system of 
functions 

A^(z) = A^,,...,^^(z) 5^(zz*) det(l - zz*r (4.12) 
where are the Schur functions, forms an orthogonal basis in the space and 

Il^-llv^ = ngT^ IJ r^(a + .,-. + i) ^'-''^ 
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Proof. First we explain the origin of this basis. Consider the model D{§{Ha, Ha) of the kernel 
representation (see 4.3). The space corresponds to the space of K- intertwining operators 
Ha — > Ha- In 3.2 wc constructed the decomposition 

Ha — ffi/iPolp,g 

into the sum of pairwise distinct representations of K. Let c{fi; a) be given by (3.6). By the Schur 
Lemma (see js^, 8.2), any K-intertwining operator is a scalar operator in each summand. Our 
basic element corresponds to the operator J^, that is c(/i; a) on Pol^^ and on Pol^ ^ for 

It remains to evaluate the Hilbert-Schmidt norm of the operator J^. By (|4.5| ), it equals 
dimPol^,^ = dimpP^ • dimp^^ ^^^ ^ 

and we apply the Weyl formula ( |3.4| ). Kl 

4.7. Comments. 1) Formula ([I.T]), ( [4.9[ ) make sense for an arbitrary matrix ball (see 2.10), and this 
defines scalar kernel representations of any classical group G, see , . 

2) General kernel representations (the definition was given in jsi], see also |^^) are realized in spaces 
of vector-valued functions on matrix balls, and analogues of the Berezin kernels are invariant matrix- valued 
positive definite kernels on matrix balls. 

We shall give the definition of the kernel representations more formally. Consider an Hermitian sym- 
metric space G/K (see 3.7). Consider a symmetric subgroup^ G C G, denote by K the maximal compact 
subgroup in G. It turns out to be that there are two possibilities. 

a) The first case. G/K is a complex submanifold in G/K 

h)The second case. G/K is a totally real submanifold in G/K and AimG/K = i AiraG/K. 

Consider a unitary highest weight representation r of G and its restriction to G. In the first case the 
spectrum of the restriction is discrete and the problem of decomposition is reduced to a purely combinatorial 
problem (Q). 

By definition, the restriction in the second case gives a kernel representation. 

For a discussion of a priori relations of this problems with other spectral problems (Howe dual pairs, 
on Stiefel manifolds, on pseudo-Riemannian symmetric spaces) see ^l| . 

3) Only 4 real exceptional groups (from 23) have kernel representations. 

4) The case G = 0(2, n) difi'ers from the cases G = Sp(27i, R), U(p, q), SO* (2n) {"matrix balls cases") 
in many details. 

5) Canonical bases exist for all scalar kernel representations (in fact, for any irreducible representation 
of K, the space of if-fixed vectors has dimension ^5 IV For matrix ball cases they consists of Jack 
polynomials (for a definition of the Jack polynomial see |39|]); in our case G = U(p, g), the Jack polynomial 
reduces to the Schur functions. Existence of the canonical bases for the case in which G/K is an Hermitian 
space was observed in ]6^ . 

5. Index hypergeometric transform: preliminaries 

We use the standard notation for the hypergeometric functions 

P / . \ (a)„(&)„a;" 



3^2 



,=0 "■(^)« 
a,b,c 1 (a)n(&)n(c)„a;" 



; X 



E 



d,e ' 1 ^ n! (d)„(e)„ 



where (r)„ = r{r -f 1) . . . (r + n — 1) is the Pochhammer symbol. 

symmetric subgroup in a scmisimple (resp. classical, reductive) group G is the set of fixed points of an 
automorphism of order 2. 
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5.1. Index hypergeometric transform. Fix b,c > 0. Consider a sufficiently decreasing 
function f{x) on the half-line R+ : x ^ 0. We define the integral transform Jb,cf by 



lis) = Jb,cf{s) 



1 



+ 00 



r(6 + c) Jo 

The inverse transform is given by 



f{x) 2Fi{h + is,b- is; b + c; -x)x''+''-\l + xf-^dx (5.1) 



1 



nb + c) Jo 



4-00 



2Fi{h + is,b — is;b + c;—x) 



T{b + is)T{c + is) 



T{2is) 



ds (5.2) 



The integral transform Ji,^c is called the index hypergeometric transform, or the Olevsky transform, 
or the generalized Fourier transform or the Jacobi transform, or the Fourier- J acobi transform, see 
see also For the first time, the inversion formula was obtained by Weyl in 1910 (||7^]). 

5.2. Plancherel formula. The following statement is equivalent to the inversion formula 



Theorem 5.1 (Weyl 0, Hi) The transform .h^c is a unitary operator 

T{b + is)r{c + is) 2 



Jb,c ■ 



[l + xf-'dx 



T{2is) 



ds , 



5.3. Exotic Plancherel formulas. Denote by W^^ the Hilbert space of holomorphic functions 
in the disc |z| < 1 with the scalar product 



1 



27rr(2a- 1) 



f{z)g{z){l - \zYY''-'2Fi (a - 5, a - c; 2a - 1; 1 - \zY){dz} 



\z\<l 



where {dz} is the Lebesgue measure on the circle. A simple calculation shows that the reproducing 
kernel of the space W^^ is 



r(6 + c) 



a + 6, a + c 
b + c ' 



(5.3) 



Theorem 5.2. The operator 

oo 

1 



|r(a + is)|2r(6 + c) 







X 2Fi{b + is,b - is;b + c; -x)x''+''-\l + x)'''''dx (5.4) 



is a unitary operator 



'(• 



'+1 



V{a + is)T{b + is)V{c + is) 



T{2is) 



ds 



(5.5) 



Remark. Let g e VF^'c- Consider the new function f{x) :— g{x/{x + 1)). The function / is 
defined on the half-pla ne R e a; > — 1 and thus we obtain a space of holomorphic functions on this 
half-plane. The kernel (5.3) is replaced by 



T(b + c) 



a + b,a + c 



xy 



b + c '{l + x){l + y) 



In formula (5^) the factor g{x) changes to f{x) (and the formula w ill a lmost coincided with (p.l|). 
The Plancherel measure on the half-line R* will be the same as in (5.5). □ 
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5.4. Index transform and Hahn polynomials. Let a,b,c> 0. The continuous dual Hahn 
polynomials (see, for instance 6.10, |^) are defined by tlie formula 



S„(s^; a, 6, c) = (a + 6)„(a + c)„ 3i^2 



-n, a + IS, a — IS 



a + 6, a + c 

The family of functions §„ is an orthogonal basis in the space 

T{a + is)V{b + is)V{c + is) |2 



(5.6) 



and moreover 

§„(s^;a, 6, c)S,„(s^;a,5, c) 



V{2is) 

T{a + is)T{h + is)V{c + is) 2 



ds 



r(2is) 



ds 



= r(a + 6 + n)r(a + c + 7i)r(6 + c + n) n! 5„i^„ (5.7) 

Lemma 5.3. The image of the function 

under the index hypergeometric transform J^^c is 

r(a + is)r(a — is) 



\ — a—b 



T{a + b + n)T{a + c + n) 
and the image of the function z" G under ^ is 

§„(s^;a,5, c) 



§„(s^;a, 6, c) 



T{a + b + n)T{a + c + n) 



The Lemma can be checked by a more or less direct calculation (see The second part of the 

Lemma implies Theorem 5.2, since we have an explicit correspondence of the orthogonal bases (z" 
and the Hahn polynomials). 

6. Helgason transform and spherical transform 

Here we discuss the spherical transform (it is also is called the Harish-Chandra transform) and 
the Helgason transform. The latter is used only for an explanation of the former. In 6.9-6.10 
we dogmatically define the spherical transform independently on the Helgason transform. Proofs 
of all facts on the spherical representations and the spherical transform formulated in 6.1-6.8 are 
contained in Helgason chapter 4. 

6.1. Definition of spherical representations. An irreducible representation in a complete 
separable locally convex space (see [Q, 7.2) p of the group G in a space W is called a spherical 
representation if W contains a K- invariant vector (this vector is called a spherical vector) . 

Under some minor natural conditions on the space and representation, a spherical vector is 
unique up to a factor (see |2^, IV. 4 and references in this book). 

Denote by ^ the spherical vector of a spherical representation p. Consider the operator in the 
space W given by 



n 



p{k) dk 



(6.1) 



K 



where dk is the Haar measure on K such that the measure of the whole group is 1. Obviously (see 
[ p4| , 9.2.1) n is the K-intertwining projection to the vector ^. 
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The vector np{g)£, has the form , where $(5) £ C is a scalar. The functfon $(.9) is called 

the spherical function of the representation p. 
Let ki,k2 eK. Then 

Hhgh)^ = npihgh)^ - np{h)pig)p{k2)^ = p{k{)np{g)i = p(fci)$(g)e = 

This implies the K x K-invariance of the spherical function: 

$(fci5fc2) = $(g) 

Hence we can consider a spherical function as 

a) a function on double coscts K \ G/K 

b) a K-invariant function on Bp ^ = G/K 

As in 2.9, we shall consider K-invariant functions on Bp^q = G/K as functions depending on 
variables Xj . Recall that the eigenvalues of z*(l — Z2;*) z. 

6.2. Construction of spherical representations. As in 2.1, consider the space © C 
equipped with the form Q having a matrix ( J ) . A subspace i? C © C is called isotropic if 
the form Q is the identical zero on R. Denote by IGr^ (the isotropic Grassmannian) the set of 
all isotropic m-dimensional subspaces in © C. The space IGr^ is a G-homogeneous space. It 
is also K-homogeneous, hence there exists a unique K-invariant measure (or a volume form) on 
IGr,„|^ For 5 € G we denote by jmig, R) the Jacobian of the transformation R i—i- gR. 

An isotropic flag in © C is a family of isotropic subspaces 

3i : i?i C i?2 C • • • C Rp 

such that dimi?j = j. Denote by Flp^^ the space of all isotropic flags in © C. The space Flp^^ 
is a G-homogeneous space. 

The stabilizer of a flag in G is a minimal parabolic subgroup in U(p, q). Denote by ei, . . . , e^+g 
the standard basis in © C. Consider the isotropic subspace Sj spanned by the vectors + Cp+i 
for i ^ j. Consider the flag 

S : 5i c 5*2 C • • • C S-p 

We define the standard parabolic subgroup P C G as the stabilizer of the flag § in G. 
The space Flp^g is also K-homogeneous, 

Flp., =K/M, where M ~ U(g - p) X U(l) X ••• X U(l) (6.2) 

p times 

hence there exists a unique (up to a factor) K-invariant measure on Flp g. For g G G we denote 
by J{g, 3V) the Jacobian of the transformation Jl 1-^ gJl. 

A multiplier uj{g,x) on a homogeneous space G/H is a function on G x (G/H) satisfying the 
condition 

^{9192, x) ^ uj{gi,g2x)uj{g2,x) 

Example. The Jacobian is a multiplier. In particular, J(g,5V) and jm{g, Rm), m = 1, . . . ,p 
are multipliers on Flp^^. 

Let Lu he a multiplier. Then the formula 

Puj{9)f{x) = f{g{x))uj{g,x) 

defines a representation of G in the space of functions on G/H. For a description of all multipliers 
in general case see Q, 13.2, see also Q, 13.5 for geometric explanations. 

^^Proof. Consider an arbitrary positive volume form Q on IGrm- Then the average of Q over the group K is an 
invariant volume form 
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Consider the action of G on the homogeneous space Flp^g. Obviously, any function of the form 

p 

m—1 

is a multipher on Fl^.g. It is more convenient to define the "basic" multiphers by the formula 

Wm(5, y) = jm(5, Pm) jm-1 (S, P„ ) " ^(,-^1™- 1, • for m = 2,...,p 

In this notation, 

m 

Fix Si, . . . , Sp e C. We define the representation ps of the group G in the space of C°°-smooth 
functions on Flp^q by the formula 

Ps{9)f{y) - f{97)J{9,y)'^'l[uj,{g,P,r^ (6.3) 

For (si, . . . , Sp) in a general position, the representation is irreducible. Otherwise there exists a 
unique irreducible spherical subquotient in pg. Let us describe it. 

Denote by M the minimal closed ps-invariant subspace of C°°(Flp^g) containing the function 
/(CP) = 1. Denote by N the maximal proper closed p^-invariant subspace in M |^ and consider 
the quotient M/N. We denote by ps the representation of G in the space M/N (if ps is reducible, 
then ps~Ps)- 

Consider the space with coordinates si,...,Sp. Consider the hyperoctahedral group Dp, 
i.e., the group generated by all permutations of coordinates and by the reflections (si, . . . , Sp) — > 
(ciSi, . . . , CTpSp), where aj = ±1. This group also coincides with the so-called restricted Weyl group 
ofU(p,g). 

Theorem 6.1. a) The representations ps are exactly all spherical representations ofG. More 
precisely, the representations ps are the all spherical Harish- Chandra modules up to equivalence of 
Harish- Chandra modules. 

b) The representations ps and ps' are equivalent iff there exists 7 € Dp such that ^s — s' . 

Denote by Gsph the set of all unitary spherical representations. 

Explicit description of the set Ggph is unknown. We shall formulate two facts about Ggph that 
are necessary for understanding of the subsequent text. 

1. Obviously, if all the coordinates Sj are pure imaginary, then the representation ps is unitary 
in L^(Flp.(j). These representations are called representations of the spherical unitary principal 
nondegenerate series. 

2. If a representation ps is unitary, then sj e RU iR (this a corollary of Theorem 6.1.b; indeed 
a unitary representation p is equivalent to its contragredient (= dual) representation, and the 
representation dual to ps is p-g). 

6.3. Another realization of spherical representations. Let h be an element of the 
standard parabolic subgroup P. Then h induces a linear transformation in each 1-dimensional 
quotient Sj/Sj-i, clearly, it is the multiplication by some complex number X? (g)F^ 

Let Sj G C. Consider the space Lg of all smooth functions on G satisfying the condition 

^^The space M is the cyclic span of the vector 1. Consider a proper G-invariant subspace L C M. Then 1 ^ L. 
Hence L is contained in the kernel of the K-invariant projection 77 (see (6.1)). Thus a sum of all proper subspaces 
in M is contained in keriJ. 

-'^'^ certainly Xjid) ^''c the diagonal elements of the matrix h in the basis ei + Sp^i, . . . , Bp + e2p, e2p+i, . . . , Cq, 
ep — e2p, ... , ei — e^+i. 
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F{rh-^) = F{r) f[ reG,heP (6.4) 

m— 1 

Denote by p the representation of G in Ls given by 

p(5)i^(r) - F(.gr) (6.5) 

Lemma 6.2. Any element of G has the decomposition g — hk, where /i G P, /c G K. 

Proof. This is equivalent to the transitivity of K on Flp_g = G/P. M 

Thus any function G is is determined by its restriction to the submanifold K C G. In ( |6.2| ) 
we defined the subgroup M := K n P. For h G M, we have |x(/i)| — 1. Thus F can be regarded 
as a function on K/M = Flp.g. 

Thus we obtain a canonical operator from Lg to the space of C°°{F\p^q), it is not hard to check, 
that this operator intertwines p with p. 

Remark. This construction also explains the appearance of the "canonical multipliers" Wm, 
they corresponds to the 1-dimensional characters |Xm(^)| of P. □ 

6.4. Preliminary remarks on Plancherel formula. 

By a general abstract theorem (see , 4.5, 8.4), any unitary representation of a locally compact 
group is a direct integral of irreducible representations. For some types of groups (including 
semisimple groups and, hence, U(p, q) = U(l) x SU(p, q)) this decomposition is unique in a natural 
sense. 

Theorem 6.3. a) Representation of G in L^(G/K) is a direct integral over Ggph with multi- 
plicities ^ 1. 

b) A kernel representation Tq is a direct integral over Ggph with multiplicities ^ 1. 

6.5. Proof of Theorem 6.3. Denote by M(G) the algebra of compactly supported (complex- 
valued) measures on G; the multiplication in M(G) is the usual convolution We define by ijP 
the pushforward of a measure fj, under the map g i-^ g^^- Obviously, fi i— s- /i^ is an involution on 
M(G): ifi * z/)° =i^°^fi° 

Denote by C(G) the subalgebra of M(G) consisting of measures invariant with respect to the 
transformations g i— > kigk2, where fci, fc2 G K. 

Theorem 6.4. (Gelfand, see 111, 5.1) 

a.) fP = for all G e(G). 

b) The algebra C(G) is commutative. 

Proof, a) It easily can be checked that g^^ is contained in the double coset K^K. Hence 

b) By a), the identity (/i * v)^ = fj,^ coincides with fi * v — v ^ fj,. Kl 

Let X- be a unitary representation of G in a Hilbert space H. For /i G M(G) we define the 
operator 

= / M:{g) d^i{g) 
Jg 

It is easily shown that * i/) = >c{p?)>c{v) (see |Q, 10.2). 

Denote by the space of all K-invariant vectors in H. The projection 77 to is given by 



77= / K{k)dk 
Jk 



Obviously, 77 has the form >?((5k), where (5k is the Haar measure on K normalized by the condition: 
the measure of the whole group is 1 . We consider this measure as a 5-measure on G supported by 
K). 
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Lemma 6.5. a) The operators >c(fi), where /i G C(G), are zero on the orthocomplement to the 
space H^. 

b) The operators >c{fi), where fi € C(G), are self-adjoint. 

c) Let R be an Q{G) -invariant subspace in . Let Z be the G-cyclic span of R. Then the 
projection of Z to coincides with R. 

d) // X is irreducible, then dim ^ 1 . 

e) If there exists a cyclic Ji-fixed vector S in H, then the spectrum of x contains only spherical 
representations and their multiplicities are ^ 1. 

Proof, a) ~ x{p. * (5k) = >i{p)n for ^ e C(G). 

b) We have x{ny = x{^°). By Theor em 6. 4. a, this coincides with 

c) Let h E R, then 

nx{g)h = IIx{g)nh = x{5-tC} x{g) x{5-tC}h = x{5k * Sg * 5i^)h S R 

d) The algebra C(G) is commutative. Any commutative family of self-adjoint bounded opera- 
tors in a Hilbert space of dimension > 1 has a proper invariant subspace. Now apply c). 

e) First let L be a G-invariant subspace without K- invariant vectors. Then S is contained in 
its orthocomplement L^ . But S is cyclic and hence H = L^ . 

Second, a cyclic representation of a commutative *-algebra has multiplicities ^ 1 (see [Q, 4.4, 
Problem 3). M 

Proof of Theorem 6.3 b) The distinguished vector is cyclic (see 4.4). It remains to 
apply Lemma 6.5.e. Kl 

Proof of Theorem 6. 3. a. There exists a unique G-invariant differential operator A of order 
2 on G/K. Consider the heat equation 

j^fit,z)^Af{t,z); t^O 

Let N{t; z, u) be the corresponding heat kernel and let At be the corresponding evolution operator, 
i.e., 

f{t,z) ^ Atf{Q,z) = I N{t;z,u)f{u)det{l-uu*)~P-'J{du} 

Fix t > {). Obviously, the function N{t] z, 0) is a cyclic vector in the (closed) image of the 
evolution operator At . Thus the representation of G in the subspace Im At has a multiplicity free 
spectrum. 

We have AtAf = At+t'- Hence IvnAt D ImAr if t < t. Thus L^(G/K) is the closure of the 
union of increasing family of multiplicity free subrepresentations 

Im Ai C Im C Imyli/3 C . . . 

Thus L^(G/K) itself is multiplicity free. 

6.6. Normalization of Plancherel measure. Let ps E Ggph be a spherical representation. 
Denote by Ws the space of the representation ps, denote by (•, •)s the scalar product in Ws, denote 
by a spherical vector in Ws, such that (^s,^s)s = 1. 

We shall consider Borel measurable functions / on Gsph such that a value of / at a point s is 
an element of the space Ws (we omit a definition of measurability) . 

Consider a Borel measure v on the set Gsph of spherical unitary representations of G, let M be 
the support of v. We shall consider the Hilbert space / Wsdi'{s) of all Borel measurable functions 
/ : s I— > /(s) e Ws satisfying the condition 

{f{s)J{s))sdi^{s) < oo 

M 
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The scalar product in this space is defined by 

[f,g]= [ {f{s),gis))sdiyis) 
Jm 

The group G acts in our space pointwise. We denote this representation [direct integral) by 
/ Psdv{s) (for details of the definition of direct integral see p^). 

In this Section we shall present the classical description of the measure Voois) on Ggph such that 
J Ps{g)dvoo{s) is equivalent to L^(G/K) and of a canonical unitary operator U from L^(G/K) to 
the direct integral / Wsdv{s). 

In the next Section we solve the same problem for the kernel representations of G = \]{p,q). 
These measures are called the Plancherel measures. 

Remark. A Plancherel measure is not canonically defined (see jsij, 4.5, 8.4). Let f3{s) be a 
positive function on the support M of the measure i'. Clearly, the direct integrals / Ps{g)dv{s) 
and J ps{g)\p{s)dv{s)\ are equivalent. □ 

For the case L^(G/K), we define a canonical normalization of a Plancherel measure by the 
following rule: the element s i-^- of the direct integral corresponds to the d -function on G/K 
supported by z ~ 0. 

Let us repeate this more formally. Consider an element s i— > a{s)^s of the direct integral. Let 
Fa be the image of s a{s)^s under the operator U^^. Then we require Fa{0) — J a{s) di^oo{s) 
for all a{s). 

This normalization is not an arbitrary rule. If we shall change it, then formulas for the measure 
Vao and for the operator U will change. It is easy to obtain a more complicated formula in this 
way, but it is impossible to obtain a simpler formula. 

Now we shall describe the map U (the Helgason transform) and the measure z^oo (the Gindikin- 
Karpelevich measure). 

6.7. Helgason transform and Gindikin Karpelevich measure. Consider the "section of 
wedge" model SWp,, of the space G/K = U(p, q)/Vip) x V{q), see 2.8. Fix s := (si, . . . , Sp) € C^', 
assume Sp+i = 0. Let Z = ^) e SWp,q. Denote by [Z]k the left upper k X k block of Z. We 
define the functions ^'s(^) by 

where 

(Ti = • • • = (Tp_i = 1; ap=q-p+l (6.7) 
Lemma 6.6. For any h in the standard parabolic subgroup P, 

*,(Z[''l) - ^,iZ) f[ |xm(/^)l'«-^+'-'"^+'^- (6.8) 

m— 1 

Proof. A calculation, see Kl 

Let / be a C°°-function on SWp.^ with a compact support. Its Helgason transform is the 
function 

F{r; si, . . . ,Sp); r e G, Sj G C 

defined by 

F{r; si,...,sp) = / /(Z)vI/,(zM) det 



1 

K 



K* 
\{L + L*) 



— uj-ry^j~^j-\-i)/^ 



n — 'nA- i 



(6.6) 



1 

K 



K* 



{dZ} 



(6.9) 
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By (6.8), the functions F{r; si, . . . , Sp) satisfy ( |6.4D , i.e., for a fixed s S C" a function F(r, s) is 
an element of the space Lg of the representation ps (see (|6.3D. 

Obviously, the pushforward of the transformation /(2') i-^ /(Z'^l) under the Helgason transform 
is given by (p^). 

Above we identified the representations ps and ps. Thus we can consider a function as a 



function on Flp g x 



Remarks. 1) Functions F{. . . ) are holomorphic in Si, . . . , Sp G C". □ 

2) For each element 7 of the hyperoctahedral group Dp there exists an integral operator on 

F\p q (which can be written explicitly) such that F(CP, 7s) — A^F{'J', s). □ 
Denote by Ep the simplicial cone 



Ep : si ^ si ^ . . . ^ Sp ^ 



Denote by lEp C its image under the multiplication by i. 

Theorem 6.7. The Helgason transform is a unitary operator 



L-'[SWp^q,dct 



1 K* 
K \{L + L* 



-| -p-q 

2 



{dZ}) i Flp,5 X (iSp), '^{s){ds}d'S' 



where {dZ} is the Lebesgue measure on SWp^^, dT is the K -invariant measure on Flp_q, {ds} is 
the Lebesgue measure on the set i'Sp and £H(s) the Gindikin-Karpelevich density 

Ij |r(i(g-p + i) + s,)|4 

n FT^^^yp 11 ('^fc-^') (6.10) 

T/ie pushforward of the transformation f{Z) i— > f{Z^^^) under the Helgason transform is given by 
FiJ>; s,,...,sp)^ F{gJ>; si, . . . , Sp)J{g, T)i/2 J| ^^-(g, P^y. 

6.8. Spherical transform: preliminary remarks. Consider the space of K-invariant 
functions / on G/K. As above, we can consider elements of this space as functions in the variables 
Xj > (see 2.9) symmetric with respect to the permutations. 

Consider the Helgason transform of a K-invariant function /. Obviously, F is a K-invariant 
function. Hence F depends only in the variables si, ... , Sp. Thus we obtain the transform from 
the space of symmetric functions in the variables xi, ... ,Xp to the space of Dp-symmetric functions 
in the variables si, . . . , Sp. 

If a function / in formula (|6.9|) is K-invariant, then we can replace the factor ^'^ by its average 



over the group K. But this average is the spherical function of the representation p^ (see |27 
4.4.3). Thus the Helgason transform on the space of K-invariant functions is given by 



F(s)= const- / f{x)<i>six)w{x)dx (6.11) 



where 



w{x)dx ■.= Y[x'^-P Y[ {xk- xi^Yldxj (6.12) 

j=i i<fc<i^p j=i 

The map f ^ F given by ( |6.11 ) is called the spherical transform. 

Remark. In some sense, the spherical transform is simpler than the Helgason transform. 
1. We have only p variables instead of 2pq variables. 
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2. The function s ^s{x) is holomorphic in s and £)p-symnietric with respect to s (by 
Theorem 6.1b). Hence for a compactly supported / its spherical transform is a Dp-symmetric 
holomorphic function. 

3. The index hypergeometric transform Jb^c for some special values of the parameters b, c gives 
the spherical transforms for all rank 1 groups (0(^», 1), U(p, 1), Sp(p, 1)). Heckman and Opdam 
[ p4| constructed the family of integral transforms interpolating the spherical transforms for simple 
Lie groups of arbitrary rank. □ 

6.9. Spherical functions and Berezin— Karpelevich formula. First we fix the notation 



det Cki '■— det 



/cii 

\Cpl 



Below all determinants are of the size p x p. 

Theorem 6.8. (Berezin-Karpelevich jsj, Hoogenboom[pl|) The spherical functions of the 
group G = \J{p, q) are given by 



$s(a;) — const 



detjaFi 



r 1 
2 



{q - p + 1) + Sj, ^{q - p + 1) - Sj 
q-p+1 



-Xk 



} 



ni^fc<i^p('*fc -^f ) ni^fc<(^p(^fc ^i) 

In order to simplify this expression, we introduce the notation 



(6.13) 



[q-p +l)/2 



6.10. Spherical transform. Thus the spherical transform for the group G is given by 



F{s) — f{s) = const 



f {x)^ s{x)w(x) dx 



.14) 



where ^(a;) is given by ( 6.12 ) and $s is given by the Berezin-Karpelevich formula. 

Theorem 6.9. Spherical transform is a unitary {up to a factor) operator from the space of 
symmetric functions with the scalar product 



if, 9) 



f{x)g{x)w{x) dx 



to the space of Dp-symmetric functions with the scalar product 



{F,G) 



F{s)G{s)mis) {ds} 



where d\{s) is the Gindikin-Karpelevich density ( |6.10 ) and {ds} is the Lebesgue 
The inversion formula is given by 

fix) = F{s) = const / F(s)$,(a;)«n(s) {ds} 



measure on 



(6.15) 



Remark. Theorems 6.7 and 6.9 coincide. The implicator 6.7 ^ 6.9 is obvious. Let us explain 
<;=. The Helgason transform is an operator from L^(G/K) to the direct integral of the principal 
nondegenerate series over the Gindikin-Karpelevich measure. We must show that this operator is 
unitary. Assume that the spherical transform is unitary. Then the Helgason transform preserves 
the scalar products {gv, w), where v, w range the space of K-fixed vectors and gv denotes an action 
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of the group on the Hilbert space. Thus the Helgason transform preserves the scalar products 
{giv,g2w) ~ {g2^giv,w) and hence the Helgason transform is unitary (since the vectors gv span 
the both Hilbert spaces). □ 

6.11. Some integrals with determinants. 
Lemma 6.10. Let fi be a measure on R. Then 

det{fk{xi)}det{gk{xi)}dfi{xi) . . .d^{xp) = n\ det\ fk{x)gjn{x)d^i{x)\ (6.16) 

, k^l k,l k^m L J 

if the right-hand side of the equation has sense. 

Proof. Obvious. Kl 

6.12. Some spherical transforms. Lemma 6.11. For functions f3i, ... , f3p on IR+ we 

define the function 

det{(3kixi)} 

^'<^' = IO^ <"^' 

Then its spherical transform is 

oo 

Qpix) = 2 2T^^*{ / f3kix)2Fi{r + si,r - si;2r;-x)x'^~'Pdx\ 

k<l 

We see that the integral in the curly brackets is the index hypergeometric transform Jr,rl3{s). 
Proof. We evaluate the integral ( 6.14 ) by Lemma 6.10. K 
Corollary 6.12. The spherical transform of the function Y\j H^j) ^■s 



^ dot 



x''^'^b{x)2Fi{r + si,r- s;; 2r; -x)x'^^Pdx^ 



Uk<ii4~sf) kA 

Our next Section is based on the following simple formula 
Theorem 6.13. The spherical transform of the function 



det 



(i-zz*)" = n-=i(i+^^r" 



IS 

riLi T{a-^{q+p-l) + Sk)Tia - ^{q + p - 1) - s^) 



(6.18) 



Below we shall use the notation 



h:= (q-p+l)/2 



Proof. We must evaluate 

det{2Fi(r + Sk,r- Sk; 2r; -Xj)} 



j^i I\l^k<l^p(-^k ■^f)I\l^m<n^pi^"^ -^'O 



P 



X {Xm - Xnf W xl ^ dxi . . . dXp 

l^m<n^p k—1 
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It is possible to apply directly Corollary 6.12, but it is more convenient to write 
By Lemma 6.11, we obtain 

oo 

^, / 2, detl /(I + x)-^+P-' (^^y\Fi{r + si,r - s,; 2r; -x)x'i-Pdx] 



By Lemma 5.3, the integrals under the determinant are the continuous dual Hahn polynomials 
§m-i- We obtain 

det{§m_i(s^;a - /i,r,r)} 



where B{a) is given by ( |6.18 ). By ( |5.6| ), §m_i(s^) = (s^)™^^ + . . . and hence the determinant in 
the numerator is the Vandcrmonde determinant. M 

6.13. Image of the canonical basis under the spherical transform. The canonical 
orthogonal basis in the space (see 4.6) in our coordinates Xk is given by 



p 



n {Xk - Xm) 

Theorem 6.14. The image of the function under the spherical transform is given by 

U T{a-h + Sk)T{a-h-Sk) dct{S^,+p_,(s|; a - r, r)} 
const • — • — = J- (6.19) 

3=0 



where §„ are the continuous dual Hahn polynomials ( |5.6| ). 

Proof. We apply Lemma 6.10 and Lemma 5.3. Kl 
6.14. Reduction of the Gindikin— Karpelevich inversion formula to the Berezin 

Karpelevich formula. Let Qf{x) be the same as above ( |6.17D . Then 



We must check the equahty 



Qf{s) = dei{Jr,rfm{sk)]lX{{sl - S?) 



By Lemma 6.10, this reduces to 



\T{r + is)\ 



4 



det{ / J,n{x)gk{x)x'' ^dx] ^ del{ I {Jr,rfm){s){Jr,r9k)is)—r-——-7^ds} 
k,m^jQ fe,mVo |r(2«s)|^ 

By Theorem 5.1, the matrix elements of these two matrices coincide. 

6.15. Comments. 1) The main facts concerning the Helgason transform, the spherical functions and 
the spherical transform are the same for all semisimple groups. But generally, the spherical functions of the 
real semisimple groups are certain multivariate special functions, they are one of the natural multivariate 
analogues of the hypergeometric functions (see [^J]). Simple determinant formulas for spherical functions 
exist only for the complex groups jl^] and U(p, g). 
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2) In general, the image of the canonical basis under the spherical transform consists of multivariate 
continuous dual Hahn polynomials or multivariate Meixner-PoUachek (for series GL„) polynomials. In our 
case, the multivariate Hahn polynomials admit a simple expression. 

Koornwinder constructed a multivariate analogue of the Askey-Wilson polynomials. All classical 
and neoclassical orthogonal polynomials in one variable can be obtained by a degeneration of the Askey- 
Wilson polynomials (see the treatise [^). Multivariate versions of (neo)classical orthogonal polynomials 
can be obtained by a degeneration of the Koornwinder construction (the basic steps were done in [^). 
In particular, the multivariate continuous dual Hahn polynomials or Meixner-PoUachek polynomials can 
be constructed in this way. 

7. Plancherel formula for kernel representations 

We preserve the notation 

r^{q-p+\)/2- h^{q + p-l)/2 

We preserve the notation ps for a spherical representation, Ws for its space, (•, ■)s for the scalar 
product in Wg, ^s{g) = ^s{x) for the spherical function of ps- 

7.1. Normalization of the Plancherel measure. By Theorem 6.3b, any kernel represen- 
tation Tq, is equivalent to a multiplicity-free direct integral / psdva{s) of spherical representations 
over some measure v — Va on the space Ggph of all spherical representations (this measure is called 
the Plancherel measure). 

As we have seen in 6.6, the measure Va is defined up to a multiplication by a positive function. 
Now we shall define a natural normalization of Va and of a unitary intertwining operator Ua from 
Tq to J psdvais)- We require the image of the distinguished vector G Va under Ua to be the 
function s i-^ ^s[^- 

We want to find the measure Va normalized in this way. To do this, we calculate the matrix 
element {7a{g)^a,^a) in two ways. We recall that this matrix element can be regarded as K- 
invariant function on G/K. A calculation in the kernel representation Tq, gives 

p 

{7a{g)Ea,Ea)=det{l-zz*r - JJi^ + ^k}-'' 

k=l 

A calculation in the direct integral gives 

(T„(5)S„,S„) = J {psig)^s,is)sdvais) = J $,(.g)di^„(s) (7.1) 

Thus we must find the measure on Gsph such that 

r ^ 

/ <i>six)diyais) = n(l + ^fc)"" (7-2) 
fc=i 

Conversely, if we have a measure Va on Gsph satisfying ( [7.2|) , then we have equality (^]^) for 
matrix elements. Hence the direct integral is equivalent to ToTi 

^^This normalization is consistent with the normalization of the Plancherel measure for defined in 6.6, since 
the limit of E,a S Vq as a — > -|-oo is the (5-function S(z) 

^^Let ^ be a cyclic vector of a unitary representation of a group G in a Hilbcrt space H. Assume we know the 
matrix element 7(g) = {p{g)£., 0- Let us explain why wo know the representation p itself. Then 

7(92"'fi) = (P(92)-V(9i)?,5> = (p(9i)€,P(92)0 

is a positive definite kernel (see 1.1) on G and after this we can reconstruct the Hilbcrt space H with the distinguished 
system of vectors p{g)£, in the usual way. 
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7.2. Plancherel formula for large values of a. 

Theorem 7.1. (Berezin [Q) Let a > q — p + I. Then the Plancherel measure is supported by 
the pure imaginary s and its density with respect to the Lehesgue measure on iMP is 



1 



— WT{a-h + Sk)T{a -h- Sk) 
J)fc=i 

p 

xn 

fe=i 



V{2sk)T{-2sk) 



n 



„2 \2 



(7.3) 



In fact, this is the product of ( |6.18 ) and the Gindikin-Karpelevich density (6.1C). 

Proof. We want to represent f{x) := n(l + ^i)~" ^^e inverse spherical transform ( 6.15| ) of 
some function F{s). It is sufficient to evaluate the direct spherical transform (6.14) for f{x). This 
was done in Theorem 6.13. This operation is correct (see [|l^) if / € n (i.e., a > g + p — 1). 
Then we consider analytic continuation. For a < h we have a pole of integrand and for a < h our 
Theorem 7.1 becomes incorrect, see below. K 

7.3. Analytic continuation of the Plancherel formula. In fact, the Plancherel formula 
obtained in Theorem 7.1 is the following identity for the hypergeometric functions 



k=i i^fc<;^p 

' det{2Fi(r + s„,r-Sm;2r;-a;fc)| X 



const ■ 



xf[T{a-h + s,)Tia -h-s,)f[ U {s^ - ^l)ds, . . . ds, (7.4) 

k—l k—1 \ / \ } l^/<m5Cp 

where r = (5— p+l)/2, h= (g+p— l)/2 (a direct verification of this formula is a nice exercise). 

The left-hand side of the equation is holomorphic in a € C. Let us discuss the right-hand side. 
The integrand has singularities if 

Rea = /i-fc; fc = 0,1,2,... (7.5) 

The F-function exponentially decreases in the imaginary direction (see, for instance, |^0j, v.l, 
(1.18.6)) 

|r(a-His)| = (27r)3|s|5-'^e^tl''l(l + 0(1)), |s| ^ c5o 

and the spherical functions of unitary representations are bounded by 1. Hence the right-hand side 
of (7^) is holomorphic in a except for the lines (7.5). 

Let us construct the analytic continuation of the right-hand side from the domain Re a > 
^(<Z + p — 1) to the whole C. 

By Lemma 6.10, wc can represent the identity (7.4) in the form 

P I 
11(1 + H {xk - xi) = const ■ _ X 

fc=i is;fc<i^p llj=o^ 3) 

poo 

X det<^ / s2("~^)F(a ~h + s)T{a - h - s) 2Fi{r + s,r ~ s- 2r; -Xk) x 



k.rn 



V^{r + s)T^{r-s) 
r(2s)F(-2s) 



ds 



} (7.6) 
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Lemma 7.2. Denote by I{a, s)ds the integral in the curly brackets in ( |7.6| ). Then the 
meromorphic continuation of I{a., s) to the whole C is given by 



I{a, s) ds + 47ry^ Cfc(a)2-Fi 



a — p + 1 + k, —a + q — k 
2r ' 



{a-h + fc)2(™-i) (7.7) 



0<fc</i-a 

where the coefficients Cfc(a) are given by 

r{2a -2h + k)T^{-p +l + a + k)r^iq -a- 



Cfc(a) 



T{2a - 2h + 2k)T{-2a + 2h - 2k)k\ 



(7.8) 



Remark. The expression (7^) has poles at the points a = p — 1 — k,p — 2 — k, . . . . Thus 
expression (7.7) has poles at the points a = p — l,p — 2, . . . . 

□ 



Im ao ■ 


J 







Proof. We shall obtain the first summand of the formula. Let Reao — h, assume Imao > 0. 
We want to construct an analytic continuation of the integral J /(a, s)ds to a small neighborhood 
of the point aQ. Our integrand has poles at the points s = ilmao. Consider the contour L 
shown on the Picture. In a small neighborhood of aQ, the expression Jj^ I{a, s)ds depends on a 
holomorphically and 



I{a,s)ds— I I{a,s)ds 



is the sum of the residues. 



Thus we obtain the analytic continuation of the right-hand side of (7.6) 
const 



1 



■det 



s^'^"'-'\F,{r + s,r- s; 2r; -Xk)d^lo.is)} 



(7.9) 



where the measure fia on the complex plane C is the sum of a continuous measure on the imaginary 
axis and (5-measures supported by the points ±{a — h + k): 



r^(r + s)T^(r ~ s) 
dfi^is) ^r{a~h + s)T{a -h- s) ^.o aw o a W 



r(2s)r(-2s) 



+ 2n 



0^k<h-a 



Ck{a)S{s ±{a + h- k)) 
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where {ds} denotes the Lebesgue measure on the imaginary axis and Ck{a) are the same as above 

Remark. The measures i^a are complex- valued if a e C. They are real- valued for a g R. □ 

Further, we apply Lemma 6.10 (from the right-hand side to the left-hand side) and obtain the 
equality 

P r 

l[{l + Xky= ^s{x)di^c{s) (7.10) 



k=l 



where 



di'a{s) = YTP — pL _ -A n {sl-sf)^ dna{si)...dfia{sp) (7.11) 

llj=0i J) i^fc<,^p 

and the measures /Iq are the same as above. 

The identity (7.10) is the required expansion of the distinguished matrix element in the spherical 
functions. 

In fact, in this formula we have integration over the family of planes having (up to the action 
of the group Dp) the form 

^'ki^,,,^k„ ■ si — a — h + ki, . . . , Sc = a — h + kc, 5^+1, ■ • ■ , Sp S iR (7-12) 

where cr = 0, 1, . . . ,p and kj are nonnegative intege rs. I f some kj coincide, then (due to the factor 
Yii^k ~ ^f)"^) the density of measure on the plane ( [7.12| ) is 0. Hence only the case 



ki > k2 > ■ ■ ■ > ka ^ 

really exists. 

By construction, our family of measures is meromorphic in a S C. Nevertheless we have a 
factor nr^^(Q! — j), which is a zero at the poles of (7.8). It is easy to show that our family of 
measures Va is holomorphic in a G C. The final formula without poles can be easily obtained and 
hence we omit them. 

7.4. Positive definiteness. 

Theorem 7.3. (HlQ) The formula (\7 .l(j{)-{frTl\) is the Plancher el formula. 



Identity ( 7.10| ) has the form (7.2) but our considerations do not imply that our measure Va is 
supported by unitary spherical representations. Hence some proof is necessary. An a priori proof 
of positive definiteness of all representations at the support of the Plancherel measure is given in 
fee . For our case G = U(p, g), also it is possible to apply Molev unitarizability results ||41| . 

Corollary 7.4. The Helgason transform is a unitary operator from the space of functions 
with the scalar product (4-11) to J psdvais)- 

7.5. Discrete spectrum. For the case a = p, the plane HJ?^ it is a one-point set. Hence the 
representation Pa-h+k^ a-h+kj, is a direct summand in Tq. These representations were subject 
of papers ||, Q, ||'. ' 

All other planes H^^ ^ correspond to direct integrals of some spherical series of unitary 
representations. 

7.6. Comments. 1) Theorem 7.1 (large values of a) was announced by Berezin for the series 
G = \J{p,q), Sp(2n,R) , SO*(2n), SO(n, 2), a proof was published by Unterberger and Upmeier in [Q. 
For other series, the problem was solved in |55|, the construction is based on the matrix B-function, which 
was constructed by Gindikin jl^] for the groups GL(n,R), GL(n, C), GL(n,E[) and by the author for other 
series of groups. The general Plancherel formula (for arbitrary a) was obtained in 

^'^a partial result was obtained in ]29| 
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2) The simple proof of the Plancherel formula given above works also for series GL(n,C), 0(n,C), 
Sp(2n,C). 

3) There exists a counterpart of our analytic continuation construction on the level of orthogonal 
polynomials. It goes at least to Wilson jTSj, the most general construction is contained in 

8. Boundary behavior of holomorphic functions and separation of spec- 
tra. 



Consider the planes ^'ki,...,k„ defined by (|7T2| ). We have a canonical decomposition 

'J'q - j PsdVa{s) = ^ j PsdVa{s) 

cr = 0, . . . , p; h — a>ki>--->kcr^O 

Our purpose is to obtain a natural realization of the summands of this decomposition. 

8.1. Restriction of holomorphic functions to submanifolds in boundary. Let ft C 

be an open domain, and let dfl be its boundary. Suppose satisfies the conditions 

1) If z G r2, A G C, and |A| ^ 1, then Xz G i.e., SI is a circle domain. 

2) If z G dn and |A| < 1, then Xz e n 

Let K{z,u) be a positive definite kernel on Q. Let K(z,u) be holomorphic in u and antiholo- 
morphic in z (and hence the space 55°[if ; fl] consists of holomorphic functions). 
Assume that the kernel K(z,u) is invariant with respect to the rotations 

K{e''^z,e-~''^u) = K{z,u) 

Theorem 8.1. ( ||59[| ) Let ^ he a positive measure supported by a subset M C dO,. Suppose that 

1) The limit 

K*{z,u)^ lim^i^((l-£)z,(l-e)u) (8.1) 

exists almost everywhere on M x M with respect to I-J, x ji. 

2) K* G L^{M X M, px ^) and the limit (S.l) is dominated, i.e., there exists a positive function 
7(z,u) G L^{M X M,fi X fi) such that 



\K{(l^e)z,{l-e)u)\i^-f{z,u) 



Then 

a) For any / G 25° [K] the limit 

i?^/(z) :=^Um /((l-e)z) (8.2) 

exists almost everywhere on M with respect to fj,, and the restriction operator is a bounded 
operator ^°[K] L^{M,^i). 

b) Let X be a bounded measurable function on M . Then the limit 



l^{f):= lim / f{{l-e)z)x{z)d^i{z) 



IM 

exists for all f G 05° [L] and l^ is a bounded linear functional on 5B°[L]. Moreover, the map X 
is a bounded operator from L°°(M,fi) to the space of linear functionals on 05° [L]. 

Remark. Generally, functions / G 05° [X] are discontinuous on the boundary. Hence they have 
no values at an individual point z G Theorem 8.1 claims that under some conditions there 
exists the operator of restriction of function to a submanifold in the boundary. 
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Remark. Statement b) of the Theorem means that the space Q5*[iir] contains distributions 
supported by the subset M C d^. □ 

In fact, the Theorem defines two following spaces of functions on M . 

The first space €°(Af, ji) consists of all fimctions on M that can be obtained by the restriction 
of / e *B°[L]. This space e°(M, n) is a quotient space of 05° [L] 

The second space €*(M, fi) is the subspace in *B*[L] spanned by the (complex- valued) measures 
x{z)ijl{z)^ where x{z) in L°°(A/, /i). The space €*(Af, /i) can be described more directly in the 
following way. We consider the scalar product 



(Xi,X2) := // K*{z,u)xi(u)x2(z)d^.{z)d^.(u) 

J J MxM 

in L°°{M, fj). Then €*(M, /z) is a Hilbert space associated with the pre-Hilbert space L°^{M, fi). 
The space €*(M, is a subspace of *B'^[iir]. 

8.2. Restriction operators in the spaces Va- We apply Theorem 8.1 to t he objects 
described in 4.2. The domain il is Bp ^ x Bp_^, the kernel K is the kernel La given by (4.3)). The 
set M is a submanifold lying in the boundary of the diagonal A: zi = Z2. 

We obtain the following statement. 

Corollary 8.2. Consider the space Va = 5B°[Lct] of functions on Bp^^ described in 4.1. Let 
M he a subset in the boundary ofBp^q, and let /i be a measure supported by M. Assume /x satisfies 
conditions 1-2 of Theorem 8.1. Then the limit ( |8.2| ) exists and the restriction operator is a 
well-defined operator 05° [Xq,] — > L^{M,^). 

Now let us consider the G-orbits Mk in the boimdary of Bp^q (see 2.6). 

Theorem 8.3. (||4^]) Let a < {q ~ p + l)/2 + k. Then the restriction operator is a well-defined 
operator from Va = ?B°[Lq] to the space L]^^{ALk) of locally integrable functions on Mk- 
The question is reduced to an estimation of convergence of the integral 

La{z, u)dl{z) dl{u) 

AxAdMkXMk 

where A is a compact subset in Mk and l{z) is the surface Lebesgue measure on Mk- I cannot 
simplify estimates of for our case G = U(p, q) and hence I omit a proof. 

Thus, we obtain the family of G-invariant subspaces (E*{Mk) C 05* [Lq] associated with the 
orbits Mk- 

8.3. Restrictions of derivatives. 

Theorem 8.4. ([^) Let a < {q — p+ l)/2 + k — I. Then the operator of restriction of partial 
derivatives of order I to Mk is a well-defined operator. 

Remark. Discrete part of the spectrum corresponds to the compact G-orbit Mq. □ 
For further discussion see |5^. 

8.4. Comments. l)The problem of separation of spectra in noncommutative harmonic analysis goes 
back to Gelfand and Gindikin [Q. Olshanskii fsi}] proposed a way, which in some cases separates one of 
the pieces of spectra. Our way differs from p^, [|61[, see also p^ ]. 

2. The condition K*{z,u) £ L^ is not necessary for existence of the restriction operator, some phe- 
nomena related to the restriction problem are discussed in 

3) The discrete part of the spectrum of corresponds to the minimal boundary orbit AIq . This part 
of the spectrum was the subject of the work jHoj . 

9. Interpolation between L^(U(p, g)/U(p) x U(g)) and L'^ (\]{p -\- q)/\J{p) x 
U(g)). Pickrell formula 

9.1. Analytic continuation of the Plancherel formula to negative integer a. Assume 



that a in the Plancherel formula ( 7.10 )^( 7.11 ) is a nonpositive integer, a = —N 
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Theorem 9.1. 



11(1 + ^^-)^ n i^k-xi)^l[r^N + j)x 

j=i i^fc<;^p j=i 



E 



mi , . ..,mp : 
-/V+p— l^mi >m2 >•" >mp ^0 



(2mj- - p + g + l)((g - p + mj)!)' 
(mj!)^r(iV + g + m^- + l)r(iV +p - m^) 



n 



{mk - mi){mk + mi + q - p + 1) I • det|2Fi (-m^ , ij - p+ l + m^sg-p+l; -a;/)| 



or 



(9.1) 



E 

mi , . ...nip : 
A''+p— 1^771 1 >m2 >" • >mp ^0 



n 



(2m, - p + 9 + l)((g-p + mj)!)' 



(rrifc - TO/)^(mfe + m; + 9 - J3+ 1)^ |> • $p 



(m, !) r(iV + g + rnj + l)T{N +p-mj) 

/i — mi — /i— mp — 1 (-^l 1 ■ ■ ■ 1 ('^■*^) 



Remark. The hypergeometric functions in the right-hand side of (9.1) are the Jacobi polyno- 
mials (see [|ol, V.2, or [Q, Chapters 2,6) 

2Fi(-m, g - p + 1 + m,; g - p + 1; -xj) = "^'Hg-p+l) „ _ 

r(9 -p + TO + 1) 

Thus the right-hand side of (9.1) can be represented in the form 



rp{q-p + i)llr\N + j) X 



E 



n 



{2mj —p^q-\- l){q — p ^ mj)\ 



mi ,. . .jTrip-. 

A^+p— l^mi >m2 >'">mp ^0 



L mjW{N + q + mj + l)r{N + p - nij) 
II (TOfc - mi){mk + mi + q - p + 1) \ ■ det{F„'-f ■°(1 - 2a;)} 



Proof of Theorem 9.1. The factor H^^o ^"^(0;— j) of (7T1) has zero of order 2p at a = —N. 
Hence a summand of the Plancherel formula can be nonzero only in the case than the product 
Ylj Ckj (a) has a pole of order 2p ai a — ~N . But only the factors 



r^(-p+l + a + fcj 



(9.3) 



of Ckj{a) (see (7.8)) can give poles (the poles of the first factor in the numerator in ( |7.8| ) are 
annihilated by the poles of the first factor of the denominator). Hence all p factors (3.3) have to 
be present, and hence the continuous components of the Plancherel measure are absent. Now 
it remains to write the formula. Kl 



36 



Our next aim is to explain the group-theoretical meaning of Theorem 9.1. 

9.2. Symmetric spaces U(p + q)/\J{p) x U(g). Consider the space 'CP C equipped with 
the standard scalar product. We denote by \]{p + q) the unitary group of this space. We write 
elements of U(p + q) as {p + q) x {p -\- q) matrices g — ( " |j ) . 

We denote by Grp^^ the space of all p-dimensional subspaces in © C^. Obviously, Gr^^q is a 
U(p + q) homogeneous space 

U(p + q)/\]{p) X U(<z) 

The graph of a linear operator C — > is an element of Grp^g, and elements of Gvp^q in a 
general position have this form. Hence we obtain a parametrization of an open dense subset in 
Grp_g by points of the space Matp,, of all p x g matrices. 

The action of the group U(p + q) on the space Matp^^ is given by the same linear-fractional 
formula as above (^). 

The U(p + q)-invariant measure on the space Matp^g is given by 

det(l + zz*)-P-«dz (9.4) 

9.3. Representations t^m of the group \]{p + q). We define these representations in three 
ways. 

1) r_7v is the irreducible representation of U(p+q) with the signature {N, . . . , A^, . . . , 0), where 
N is on the first p places (see §49). 

2) Denote by Det the determinant line bundle on Grp^g. Denote by Det^^ the A^-the power of 
Det. Then t-n is the representation of U(p -I- q) in the space of holomorphic sections of Det'^^ . 

3) Consider the space ^°[L-n] of functions on Matp,g defined by the positive definite kernel 



L_pf{z, u) = det(l -|- uz* 



Lemma 9.2. The kernel L^n is positive definite. 

Proof. By Proposition 1.6, it is sufficient to prove the statement for = 1. Consider 
Euclidean space C^+* with an orthonormal basis ei, . . . , Cp, hi, . . . , hq. Consider the p-the exterior 
power A^C^^'' and the system of vectors 

Vz := (ei + ^ zijhj) A (ea + ^ Z2'jhj) A • • • A (ep + ^ Zpjhj) G APCP+« 

Then (w^,i;„) = det(l -I- zw*). Kl 
The functions 

are holomorphic polynomials of degree ^ pN. By Corollary 1.4, all elements of the space QS° [i-w] 
are holomorphic polynomials of degree ^ pN. 

The group U(p -I- q) acts in *B° [L^n] by the operators 

r-N{9)f{z) = f{{a + zc)-\b + zd)) det(a + zc)^ 

Remark. The image of an element Ou{z) :— det(l -I- zu*)^ of the supercomplete system under 
the transformation T-N{g) is an element of the supercomplete system (up to a factor). Hence the 
operators T-^ig) preserve the space 5B°(L„Ar). □ 

9.4. Kernel representations of \J{p + q). Consider the positive definite kernel 

, , , |det(l + uz*)|2^ 



det(l + zz*)^ det(l + uu*)^ 



and the space V-n '■= ^°[^-n] defined by this kernel. We define the kernel representation 7^n 
of the group U(p + q) in the space Q5°[L-jv] by the formula 

7.N{9)f{z) = .f{{a + zc)-\b + zd)) 
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We also define the distinguished vector 5_Ar. It is the function /(z) = det(l + zz*)^^ . The 
orbit of the vector S_jv consists of ah elements of the supercomplete system L-n{z, a) and hence 
the vector S_Ar is cyclic. 

The same arguments as in 4.3 show that the representation T_jv can be decomposed into a 
tensor product of t^n and the complex conjugate representation r_jv. Also the space Q3°[£;_7v] 
can be canonically identified with the space of operators Q5°[L_jv] ^ ^°[L-n]- 

9.5. Why formulas in Sections 4 and 9 are similar? Representations (see 3.1) make 
sense for arbitrary complex a. For arbitrary real a the kernel Ka{z,u) = det(l — uz*)""' defines 
some scalar product in a space of holomorphic functions, but this scalar product is not positive 
definite. The work in such spaces is possible but it is difficult from analytical point of view. 

For a negative integer a our representations arc finite-dimensional and hence they can be 
extended holomorphically to GL(p + g, C). It is more natural to consider them as representations 
of the compact form U(p + q) of the group GL(p + q, C). 

In fact, the formulas of Sections 4 and 9 for actions of groups and scalar products really coincide 
and the reason of small difference in signs is the jump to another real form of the group GL{p-\-q, C). 

9.6. Plancherel formula. We say that an irreducible representation of \]{p+q) is U(p) x U(q)- 
spherical if it contains a nonzero U(p) x U(<7)-fixed vector. By the same Gelfand Theorem 6.4, this 
vector is unique. 

The U(p) X U(g)-spherical functions of U(p + q) are given by the same Berezin-Karpelevich 
formula^. 

Lemma 9.3. The kernel representation T.^v is a multiplicity free sum of the spherical repre- 
sentations. 

Proof. This is almost a special case of Theorem 6.3b. Nevertheless we shall give an indepen- 
dent proof. 

First the vector S_jv is cyclic and U(p) x U((7)-invariant. Hence the orthogonal projection of 
S_7v to any subrepresentation is U(p) x U(g')-invariant and cyclic in the subrepresentation. Thus 
all irreducible subrepresentations of T-n are spherical. 

Secondly, assume that T-n contains two copies of the same irreducible representation p' of 
\J{p + q). Let V, V' be the corresponding orthogonal subspaces and let ^ G V, ^' V be the 
spherical vectors. Let J : V V he the unique intertwining linear operator V ^ V such that 

— ^' . The projection of S_jv to V (BV must have the form /i^', where A, /i G C. But this 
vector is not cyclic mV®V' (since it is contained in the graph of the operator jiX^^J : F — > V'). 
Kl 

The definition of the Plancherel measure given in 7.1 is valid in our situation. But direct 
integrals here reduce to finite sums and we shall give the definition again. 

Let /O^ be the spherical representations that occur in the spectrum of T_jv, let be the spherical 
function of , and let be the projection of the distinguished vector S-at to the subspace VK 
The Plancherel measure in our case is the collection of numbers 

The matrix element of the distinguished vector can be expanded into the sum of spherical functions: 
As a result, we obtain 



Theorem 9.4. The Plancherel coefficients are the coefficients in formula (3.2). 

9.7. Limit as — ^ oo. Consider the space ?8*[iL_Ar] (see 1.6). Consider measures on Matp ,j 
of the form (^(z) det(l + zz*)^^~'^\dz\^ where are compactly supported smooth functions. Then 

■^^In this case, —1 ^0 are the eigenvalues of the matrix (— z*(l -t- zz''')z), where z £ Matp^q. 
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we obtain a scalar product in the space of smooth compactly supported functions given by 

{du} {dz} 



^ ' JJ ^ ' '^^ '^^ Met{l + zz*)P+9det(l + uu*)P+9 ^ ' 

Matp,gXMatp_, 



It is natural to choose the normalization constant by 

d-N)^ [ {1 + zz^y^-P^^dz} 

Proposition 9.5. The limit of scalar products (|9.5|) as N ^ +oo is the -scalar product with 



respect to the \J{p + q)- invariant measure (9.4). 
Proof is similar to the proof of Theorem 4.3. 

9.8. Pickrell formula. 

Theorem 9.6. Let us omit the condition N + p — 1 ^ mi in the summation in (9.2). Then 



formula ( |9.2[ ) remains valid for all complex numbers N such that Re > — 1 . 

For p = q this gives the Pickrell formula |^ (in this case, the Jacobi polynomials arc the 
Legendre polynomials). 

Remark. Let N be a positive integer. Then omitting of the condition N + p — 1 ^ mi does 
not change formula ( |9.2|) . All new summands, which appear in the formula, are zeros, since we 
have the factor r(A^ + P — rni) in the denominator. □ 

Theorem 9.7. (Carlson, see [0, 2.8.1) Let f{z) be holomorphic function for Rcz > 0, let 
f{z) = 0(e'''^~'^-"^'), and f{n) = for all integers. Then f{z) is identically zero. 

Proof of Theorem 9.6. A simple calculation with Lemma 6.10 shows that the functions 
'^p~h-mi-i....,p-h~mj,~'i{xi, ■ ■ ■ ,Xp) Constitute an orthogonal system in the space L^ of symmetric 
functions on the cube ~l ^ Xj ^ with respect to the measure 

da{x) := JJ(-a;fe)''"^ JJ(a;fe - xi)'^dxi . . . dxp 



Hence the coefficients of the expansion of J|(l + Xj)^ in the spherical functions are L^-scalar 
products 



Cm{N) — const (m 



/ TT(1 + Xk)^ ^p-.h-7ni-i,...,p-h-nip-i{x) da{x) 



Obviously, these coefficients are bounded for Re iV ^ 0. Since 

T{N + a)/T{N + h) N"-^, N 



(see |30|, v.l (1.19.4)), it follows that the coefficients Cm{N) of (9.2) also are bounded. Now we 
apply the Carlson theorem to c„i [N) — Cm {N) . M 
9.9. Comments. 1) The formula (|9.2|) itself is an extension of the Pickrell formula |^^. Nevertheless, 
our method of analytic continuation (seeEtt]) is the same for all symmetric spaces and the decompositions 



of the type (3.2) follow automatically from the B-integrals evaluated in |S4[ . 

2) The limit of formula (^^) as ^ +oo gives the Plancherel formula for on the compact symmetric 
space L^{\]{p + q)/\J(p) x U(g)). 

3) In particular, this gives the Helgason theorem on the description of spherical representations of 
compact groups, see j25|, ji^. Theorem 5.4.1. 

4) As in Subsection 4.6, we obtain a canonical basis in the subspaceV^jv of K-invariant functions 
in V_jv (since V?5jv is equivalent to the space of operators in 23°fL_jv). Its image under the spherical 
transform consists of determinants of dual Hahn polynomials (see |3q| on the dual Hahn polynomials). 
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For other series this gives multivariate dual Hahn polynomials or (for series GLn) the multivariate 
Krawtchouk polynomials. 

5) There exists a natural inverse (!) limit of the symmetric spaces 

lim U(p + fc + p)/U(p) X U(fc + p) as p ^ +00 

It was constructed in the important work of Pickrell |33] (for fc = 0). Pickrell's type construction exists 
for all classical compact symmetric spaces (|^^, [Q, Q)- The are many similarities between harmonic 
analysis on the inverse limits of symmetric spaces and the analysis of Berezin kernels (see |^^, |5^], [ p7[ , 
0)- 

10. Radial part of the spaces and Gross— Richards kernels 

10.1. An orbital integral. Consider the space consisting of K-invariant functions / G V^, 
see 4.6. 

Proposition 10.1. The space has the form Q3°[i?Q,], where the reproducing kernel Ra{z,u), 
z,u € Tip q, is given by 

R4z,u) = detil-zz*)"det{l~-uu*)°' J \det{l-kizk2^u*)\-^°'dkidk2 (10.1) 

kiGU(p),k2eU(<7) 

where dki, dk2 denote the Haar measures on the unitary groups V{p), U(q) such that the measure 
of the whole group is 1. 

Remark. The kernel Ra{z,u) is invariant with respect to the transformations z 1-^ ki2rk^"'^, 
u 1-^ liwl^^, where ki,li G U(p), k2,l2 G U(g). Hence all elements of the space ^°[Ra] are 
U(p) X U(g)-invariant functions. □ 



Proof. Let a G Bp g. We want to find a function Xa G such that for any / G 



f{a) = {f,Ma)v^ (10.2) 

Let 

^ , , det(l - aa*)"det(l - zz*)" 
e^z) = Ii.iz,a) . |det(l-.a*)r 

Then, by the reproducing property (|l.3| ) 

/(a)-(/,ea)v„ (10.3) 

This implies boundedness of the linear functional / t—f f{a). Thus we can apply 1.3. Thus the 
reproducing kernel exists. 

The function / is K-invariant and hence we can replace 9a in equation (10.3) by its average over 
the group K. This gives the integral expression (10.1) for the function >Ca and for the reproducing 
kernel Ra{z,u) of our space. M 

10.2. Evaluation of the reproducing kerneL Let us write functions / G as functions 
depending of the variables xi, . . . ,Xp, see (2.9). 
Theorem 10.2. 

(q - p)\P n^^i r2(a - J + 1) 



nKfe<i^p(a;fc - ^i) nKfe<i^p(yfc - yi) 



(10.4) 
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Remark. The kernel (10.4) is a special case of the Gross-Richards kernels □ 

Theorem 10.3. (S. Bergman, 1947) Let K be a positive definite kernel on a set X. Let 
Ci(a;), ^2(2^), ■ • ■ be an orthonormal basis in the space 05° [if]. Then 

K{x,y) ^YM^'^oiv) 
3 

and the series in the right-hand side converges on X x X . 

Proof of Theorem 10.3. Consider the expansion of the function 9a{x) := K{a,x) with 
respect to the basis Qj 

ea{x) ^Y.^,{a)Q{x) (10.5) 

This series converges in Q5°[i4r] and hence it converges pointwise. Let us evaluate {da{x),Ck{x)) in 
two ways. By (10.5), it is equal Ck{a). By the reproducing property (1.3), it equals to Cfe(o)- Kl 

Proof of Theorem 10.2. We must evaluate 



where is the canonical orthogonal basis in defined by ( 4.12 ). This reduces to the evaluation 
of the series 

E ri (^/+p"-^j)!V/+V-j)!'""---'"'^^'' ■ ■ ■ ' ^p^'^^'-''^"^^'^ ■ ■ ■ ' ^^'^ 

where 

Xu = X},l{\^ xu), Yk = yk/{l + yk) 

and are the Schur functions. 
Theorem 10.4. (Hua, H) 

det{^ a^X-Y,"} = °- ■ ■ ■ det{X,"^} det{F;^} 

Ti^O Tii>n2>--->np^0 

This can be verified directly. Let us write this identity in the form 

det{E„^oan^r>?} 
n {X,-X,) n {Yu-Y,)= E a,,^v-^a,,+r>-^...a,^s,{X)^,{Y) 

Now Theorem 10.2 becomes obvious. Kl 

10.3. Spherical transform in the spaces V^. 

Proposition 10.5. Let a > h = ^{q +p — 1). Then the spherical transform is a unitary 
operator from to the space L^ with respect to the Plancherel measure (7.1). 

This is a rephrasing of Theorem 7.1. This is also a simple corollary of the exotic Plancherel 
formula (see 5.3) for the index hypergeometric transform. 

10.4. Comments. 1) Analogues of the orbital integral ( 10.1 ) (the average of the Berezin kernel over 
the compact subgroup) exist for all matrix balls. 

2) The calculation of 10.2 is valid also for the series Sp(2n, R), S0*(2n), GL(n, C). The last case was 
considered by Gross and Richards I24]. 
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3) For other matrix ball series, the Schur functions in (10.6) are replaced by Jack polynomials. I cannot 
identify these kernels with some known special functions. It seems, that they define "new" scalar products 
in the space of symmetric functions. 

11. 0rsted problem. Identification of kernel representations and L^(G/K) 

11.1. The problem of unitary equivalence. As we have seen, for a > {q + p — l)/2 the 
Plancherel measure for Tq differs from the Plancherel measure for L^(G/K) by a functional 
nonvanishing factor ( |6.18 ). Hence ioi a > {q + p — l)/2 the representation Tq, is equivalent to the 



representation of G in i (G/K). 

It is easy to construct an intertwining operator from Tq, to L^(G/K). The simplest possibility 
is to consider the identity map / ^ /.Q 

There arises a problem of construction of an explicit unitary operator 

Jq : L^G/K) ^ Vq (11.1) 

11.2. A-function. Our construction is based on one special function. Let a, 6, c > 0. Following 
[|7|, we define the A-function by 

CX) 

1 f ^{b + ^snb-^snc + isnc-^s) 







X 2Fi{b + is, b — is;b + c; —x) ds 



It seems that A-function cannot be expressed in terms of the standard special functions by algebraic 
operations (except some special values of the parameters b,c). I discussed this function in detail 
in 0. 

11.3. Construction of the unitary intertwining operator. We preserve the notation 
for the distinguished vector in Vq, (see 4.4) and the notation fiaiz, u) for the reproducing kernel of 
the space Vq,. 

Suppose we know the function 5 = JqT G L^(G/K). For g — the image of the 

function 

S„(zM)=i:^(^,a-i5) 

under is ^{z^^^). Hence if we know J~^Sct, then we know the J~^-image of all supercomplete 
system 9u{z) = La{z,u). We define the unitary G-intertwining operator : i^(G/K) Va by 
the formula 



Jc.f{9)^{f,d{z^'^))L-= / f{zmzi9])det{l-zz*r'"Hdz} (11.2) 



The distinguished vector Sq, is K-invariant, hence the function ^{z) also is K-invariant. Thus the 
function J^/ is a left K-invariant on the group G and hence Jq/ is a function on G/K. 
Now we want to find a K-invariant function J from the equality 

(^(z),^?(zM))i.(G/K) = (S(z),S(zM))v„ 



^■^ Scalar products in and L^(G/K) are different and problem of houndodness of the iden tity map is nontrivial. 
This identical operator Id was discussed in many papers (see W, [M, K§, teg). In M there was announced 
hound edness of Id for a large a. Clearly, boundedness follows from the explicit Plancherel formula (the expression 
( p.lq ) is bounded for a fixed a; this gives also an explicit formula for the norm of Id. On the other hand, the 
Plancherel formula for vector-valued kernel representations is not known, and hence a priori proofs of boundedness 
of Id preserve sense. 
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or in explicit form 

J diz)^{ia + zc)-i(6 + zd)) det(l - zz*)-P-''dz = det(l - uu*)" = + Xfe)"" (11.3) 



where u = 0^^^ = a~^5 and the same as above ( |2.9D . 

Theorem 11.1. The function ^{x) given by 



m = ^aM^) det{4-^--;^.^-+^-^A-]-^._,(.,)} 



is a solution of the problem (11.3) and hence the G -intertwining operator Ja given by (11.2) is 
unitary. 

Remark. This solution is not unique. I think that it is the 'best' of possible solution (but this 
can be regarded as my own opinion, this subject is discussed in |5^). □ 

Let B he & p y. p matrix, let Xj be its eigenvalues. We define the K-function of the symmetric 
space G/K = U(p, g)/U(p) x U(g) by 

Formula ( |ll.2[ ) written in an explicit form gives the following statement. 

Corollary 11.2. The unitary G -intertwining operator Ja '■ £^(G/K) Va is given by 



Jcfiu)^ j f{z)k''Gf^{Q{z,u))Aet{l-zz*)-P-''{dz} 

where 

Q{z, u) (1 - zz*y^{z - u){l - u*u)-^{z* - u*) 

Proof of Theorem 11.1. Let be the spherical vector of a spherical representation ps 
acting in the space Ws (as above). 

Consider the decomposition J Ws ^{s)ds of L^(G/K) into the direct integral. The function ^ 
is K-invariant. Hence the image © of in the direct integral is a function of the form s i-^ '~f{s)£,s, 
where 7(5) is the spherical transform of ^. Under the actio n of a n element g € G the function (S 
transforms to s 1-^ l{s)Psig)£,s- Hence the left-hand side of (11.3) coincides with 

his)\'{Cs.Psi9)is)wMs){ds}^ l\^{s)\HsigMs){ds} (11.4) 

But the right-hand side of ( |ll.3| ) is the matrix element of the distinguished vector Sq,. Hence 
|7(s)p is the spherical transform of ]^(1 + Xk)^°' , and hence |7(s)p is given by ( 6.18| ). 
Now we assume 

= T f{V{a-{p + q- l)/2 + s^) 

(recall that s is imaginary). It remains to evaluate the inverse spherical transform of 7(5). This 
means that we must evaluate the integral 

/ WT{a - h + isk) det{2i^i(r + zsm, r - is™; 2r; -a;fe) > X 

/ k.m ) 

^ 1 1 



^ T\r + isk)T\r - isk) 



ni -r tsfc;-L \i - it,k TT / 2 2 NT , 



' ' V{2iSk)V{-2isk) ^ 

k—1 l^l<m^p 
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The function T[a — h + is) is not even, hence we cannot change domain of integration to W. 
Nevertheless the integrand is symmetric with respect to Sk and we can replace the integral by the 
integral 

^ / 

si^O,...,Sp^O 

(with the same integrand). 

Let us convert the last factor of the integrand to the form 

Y[{sf ~ Sm) = det{ (6 + isk)j-i{b ~ isk)j-i} 
where (b + is)j^i is the Pochhammer symbol. By Lemma 6.10, the integral reduces to 



det< / T{a — h + is) 2Fi{r + is,r — is; 2r; —Xk) X 


r(r + j — 1 + is)T{r + j — 1 — is)T(r + is)T{r ~ is) ^ 
T{2is)T{-2is) J 

Now we apply the formula (see [Q, v.l, 2.8(22)) 

h)ky''-\F,ia, /?; 7; y) = ^y^+'=-^2Fi(a, /3; 7 + fc; y) 

ay'' 

and obtain 

00 

const • det< / Tia- h + is) 2Fi{r + is,r - is;2r + j - 1; -Xk) x 

k,j L dx^ ^ J 







r(r + j - 1 + zs)r(r + j - 1 - is)T{r + is)r{r -is) \ 



T{2is)T{-2is) 

This completes the proof. 

11.4. Hidden overgroup. The operator Ja identifies the spaces and i^(G/K). We have 
seen in 4.2 that the action of G in Va extends to the action of the group G = G x G containing 
G as the diagonal subgroup. If we believe that our operator J is canonical, then we obtain the 
following strange statement. There exists a natural one-parametric family (depending of a > h) of 
actions of the group G x G m the space L^(G/K). 

11.5. Comments. 1. The problem of explicit unitary identification goes back to the work in 
this paper there was called attention to the coincidence of spectra of L^{G/K) and restriction of a highest 
weight representation of G to G. 

2. We define a A-function of a classical Riemannian noncompact symmetric space G/K by 

J^g/k{x)^ [Y[r{a + isk)^six)iR{s)ds 

•' k 

where $s are spherical functions of G and SH is the Gindikin-Karpelevich density. For six series of the 
classical groups the function Aq^j^ can be evaluated explicitly. 

3. The case G — GL(g,C), GL(g, M), GL(g, H) is trivial. For definiteness, consider the kernel repre- 
sentation of GL(g, C). It is the restriction of the representation Tq (see (3.2)) of the group \J{q,q) to the 
subgroup GL(n, C) (see lists in js^). The symmetric space GL(g,C)/U(g) is the cone PoSq defined 
in 3.6. The Laplace transform is a unitary GL(g, C)-intertwining operator from L^(PoSq) to the space 
Ha(Wq) of the kernel representation. 
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4. The calculation given in 11.3 is valid also for the series G — 0{n, C) and Sp{2n, C). 

Addendum. Pseudoriemannian symmetric spaces, Berezin forms, and 
some problems of non harmonic analysis 

Molchanov (see [Q, ^]) introduced some types of representations of certain groups G 

("canonical representations") related to expressions similar to Berezin kernels (4.7). I shall try to give a 
general scheme for pseudo-Riemannian symmetric spaces, including kernel representations together with 
Molchanov's, van Dijk's, and others examples of "canonical representations". I shall also try to discuss 
similarities and differences of these constructions and the kernel representations (A. 5- A. 8). 

In A.9-A.10 another (essentially different) problem of harmonic analysis on pseudo-Riemannian sym- 
metric spaces is considered. 



We need uniform models of all the classical pseudo-Riemannian symmetric spaces obtained in 




First we consider some examples and after this we describe the general construction. 

A.l. Several examples. In all examples below a point of a symmetric space G/H is an ordered pair 
of linear subspaces W, F in a fixed linear space V such that V — W (BY . 

Example 1. The space GL(n, R)/GL(fc, R) x GL(n — k, K) consists of pairs (W, Y) of subspaces in R" 
such that dim W = k, dim Y = n-k,9." = WS)Y. 

Example 2. Consider the linear space C^" equipped with the operator J of the complex conjugation 

J{zi, . . . , Z2n) = {Zl, . . . , Z2n) 

The group G of all linear operators commuting with J is GL(2n, R). A point of the symmetric space 
GL(2n,R)/GL(n,C) is a pair of subspaces {W,Y) in C^" such that JW = Y (and hence JY = W) and 

c^" = w e Y. 

Example 3. Consider the linear space R^" = R" ® R" equipped with the operator J with the matrix 
(o The group of all operators in R^" commuting with J is G = GL(n, R) x GL(n, R). Consider the 
set 6 of aU pairs of subspaces {W, Y) in R^" such that JW = Y (and hence JY = W) a.nd W eY = R^". 
Obviously, 

6 = G/H = GL(n,R) x GL(n,R)/GL(n,R) 

where the subgroup H = GL(n, R) is the diagonal subgroup in G. 

Example 4. Consider the linear space C © C equipped with the Hermitian form Q with the matrix 
( -?i ) a-s above (Section 2). Consider the set 6 of all linear subspaces W C C (BC such that dim W — r 
and the form Q is nondegenerate on W. Obviously 

e= U U(p,g)/(U(s,t) xU(p-s,g-t)) 

s ,t:s-{-t — r,s^p^t^q 

(this construction includes the construction 2.4 above). Let us also introduce subspaces Y which are the 
orthocomplements W''^ to W. 

Example 5. Consider the space R^" = R" ® R" equipped with the skew-symmetric bilinear form with 
the matrix ( i'j J ) . Consider the space (3 of pairs {W, Y) of maximal isotropic subspaces in R^" such that 
R^" = W ffi y. Obviously, 

6 = Sp(2n,R)/GL(n,R) 

Example 6. Consider the space R^" = R" ® R" equipped with the skew-symmetric bilinear form B 
with the matrix ( _?i J ) and the symmetric bilinear form Q — ( J _?i ) . Consider the space & of pairs ( W, Y) 
of maximal _B-isotropic subspaces in R^" such that R^" — W (B Y and Y is the orthocomplement of W 
with respect to Q. It can easily be checked that 

n 

6 = y GL(n,R)/0(r,n-r) 

A. 2. Uniform construction of classical pseudo-Riemannian symmetric spaces. In all exam- 
ples above a point of a pseudo-Riemannian symmetric space G/H is a pair (W, Y) of transversal subspaces 
satisfying some simple conditions. There are 3 types of conditions: 

a) W, Y are maximal isotropic subspaces 
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b) Y is the orthocomplement of W 

c) y = JW, W — JY for some fixed operator J. 

For a uniform description of all 54 series of pseudo-Riemannian classical symmetric spaces (see Berger 
classification, we must fix some notations and definitions from linear algebra (this general construction 
is not necessary for understanding Subsection A3-A10 below). 

The term linear space below means a right finite-dimensional module V over R, C or the quaternions 

e. 

A semiinvolution J is a linear or antilinearf^ operator in a linear space V satisfying the condition 
J2 = ±ljf| A semiinvolution J in V is split if there exists a subspace W such that V — W (B JW. 

The term form below means a nondegenerate form on a linear space V over IK = R, C, EI of one of the 
following types 

a) a symmetric or skew symmetric form over R 

b) a symmetric, skew symmetric or Hermitian form over C 

c) an Hermitian or anti-Hermitian^ form over EI 

A linear semiinvolution J is consistent with a form B(-,-) if B{Jv, Jw) — ±B{v,w). An antilinear 
semiinvolution J is consistent with B if B(Jv, Jw) = itB{v,w). 

A subspace C V is called isotropic with respect to a form B{-, ■) if B{w,w') — for all £ W. 

A form B on y is split if there exists an isotropic subspace W <ZV such that dim W = i diml/. 

A Grassmannian is a set of all subspaces of a given dimension in a linear space or a set of all isotropic 
subspaces of a given dimension. 

A real classical group is a group of all linear operators in a linear space over K (i.e., GL(n, R), Gh{n, C), 
GL(n, H)) or the group preserving some form in a linear space over K, i.e., 

- the groups 0{p,q), Sp(2n,R) over R 

- the groups 0(n, C), Sp(2n, C), U(p, q) over C 

- the groups Sp(p, </), S0*(2n) over H. 

A classical pseudo-Riemannian symmetric space is a homogeneous space of the form G/H, where G 
is a classical group, and a subgroup H is the set of fixed points of some automorphism ct of G satisfying 
= 1. 

All classical symmetric spaces (up to covering and center) can be obtain in the following way. 
Consider the following types of data on a linear space V over the field K. 
STRUCTURE 1*. A basic form B is a split form on V. 

STRUCTURE 2*. A control semiinvolution J is a split semiinvolution on V. 
STRUCTURE 3*. A control form is arbitrary form on V. 
Consider ordered pairs [W, Y) of subspaces of V such that 

V = W(BY (A.l) 

We say that a pair {W, Y) is consistent with the basic form B if the both subspaces W, Y are B-isotropic. 
We say that a pair {W, Y) is consistent with the semiinvolution J if JW — Y. 

We say that a pair {W, Y) is consistent with the control form C if y is the orthocomplement to W with 
respect to the form C. 

We consider a linear space V without any additional structure, or a linear space V equipped with one 
of the structures 1* — 3*, or a linear space V equipped with all structures 1* — 3*. In the last case we 
assume J is consistent with B and 

C{v,w) = B{Jv,w) 

Lemma A.l. // a pair [W, Y) is consistent with two of the structures 1* — 3*, then it is consistent with 
the third structure. 

Fix a linear space V with such structure. Denote by GiV) the groups of all linear operators in V 
preserving all structures on V . Consider the set S of all pairs {W, Y) such that 

1) W®Y = V 

2) (W, Y) is consistent with the structure of V 

A map A : y — > V is called an antilinear operator if A(v + w) = Av + Aw, Afiv = JlAv for all i), to £ V, ^ S K. 
^■'Tliis definition is adapted to our field K = K, CH, for a general definition of semiinvolution see jl^ . 
^^A sesquilinear form B(v,w) is called anti-Hermitian if B(w,v) = —B{v,w). 
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3) dim Y is fixed 

Observation A. 2. |^ a) If the set & is nonempty, then it is a pseudo-Riemannian symmetric space 
G{V)/H or atheunion of a finite collection of pseudo-Riemannian symmetric spaces G{V)/Hj 

b) Each pseudo-Riemannian symmetric space can he obtained in this way. 

Tables are contained in The examples given above give a representative sample. 

A. 3. Matrix atlas. Consider a symmetric space G/H obtained in this way. Fix a point [W ,Y) £ 
G/ H. Then for [W, Y) G G/H in general position, the subspace W is a graph of an operator A : W Y 
and F is a graph of an operator B : Y W. Thus for any point of G/H we obtain a pair of operators 

A:W ~*Y; B:Y 

Thus for any point {W , Y) we obtain a coordinate system (^4, B) on G/H. 

Example. For the spaces U(p, (ir)/U(p) x U(g) we obtain the Cartan matrix ball realization. 

Remark. The matrices A,B are not arbitrary. For instance in Example 1 of A.l a pair [A,B) satisfy 
the unique condition det(l — y4_B) 7^ 0. In Example 5 we have the same condition and also A = A*, B = _B*. 
Generally the pair {A, B) ranges in an open subset in some linear subspace in space of pairs of matrices. 

A. 4. Hua Loo Keng double ratio. Fix two pairs of subspaces (Wi, Yi), {W2, Y2) € G/H in general 
position (we also assume dim Wj ^ dim Yj ) . Then W2 is the graph of some operator R : Wi Y\ , and Y2 
is the graph of some operator S : Yi ^ Wi . We define the Hua double ratio operator 

Vi; W2, Y2) := SR: Wi-^ Wi 

In matrix coordinates, this operator is given by the formula 

'D{Ai,Bv,A2, B2) = (1 - B2A^y\B^ - B2){1 - A2Bi)-'(Ai - A2) 

Lemma A. 3. a) 1 is not an eigenvalue 0/ D(Wi, Yi; W2, ^2). 

b) 1 - D(Ai,Bi;^2,B2) = (1 - B2^i)"'(l - B2A2)(1 - B^A2)-\\ ~ BiAi) 

c) D/(l - D) = (1 - BiAr)-\B2 - - A2B2y\A2 ~ Ai) 

Proof. Statement a) is equivalent to W2 n 1^2 = 0, and b), c) can checked by a simple calculation. 
A. 5. Berezin form. Consider an arbitrary character x of the multiplicative group of K (for instance, 
x{z) ~ kl"). Consider the kernel 

L^{Wi,Yi;W2, Y2) = x(det[l - D(Wi, Yi; IV2, ^2)] ) 

We define the Berezin form on the space of smooth compactly supported functions on G/H by 

{f,9)~ff LAWi,Yv,W2, Y2)fi{Wu Yr)f(W2, Y2) d/i(M, Fi) dii{W2, Y2) (A.2) 

J J G/HxG/H 

where {Wi, Yi), (W2, Y2) are points of the symmetric space and ^ is the G-invariant measure on G/H. 

A. 6. Comparison of Riemannian and pseudo-Riemannian cases. Formally the construction 
A. 5 in the Riemannian case gives kernel representations. But a serious divergence between the Riemannian 
and pseudo-Riemannian cases appears immediately. 

I emphasis that the kernel is smooth on the diagonal of G/H xG/ H but (for nonRiemannian case) it 
has singularities outside the diagonal. It seems (but not proved carefully) that our Hermitian form always 
(except for the Riemannian case) is indefinite. This leads to serious technical difficulties. In particular, an 
indefinite Hermitian form does not define a topology in a functional space. So even the formulation of the 
specral problem is not obvious, for a discussion see A. 7, A. 8 below. 

Many other phenomena existing for the kernel representations do not survive in the pseudo-Riemannian 
case. For instance, there is no realization in holomorphic functions, there is no overgroup G described in 
4.7 and Section etc. It seems that theory of the kernel representations cannot be a special case of 
pseudo-Riemannian theory. 

^^G has no relation to the overgroup G° described below 
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A. 7. Indefinite harmonic analysis: approach related to Krein structures. Consider a linear 
space X equipped with an indefinite Hermitian form Q. Fix subspaces X+, X_ in X such that the form 
Q is positive definite on X+ , negative definite on X- and X = X+ X- . Let X+ be the completion of 
the pre-Hilbert space X+, let X- be the completion of the prehibert space X- with respect to the form 
(— Q). Thus we obtain a topological vector space X := X+ © X- equipped with the form Q and with the 
fixed decomposition X+ © X_ (these datas are called the Krein structure, j^). 

A Krein structure is not canonically determined by the space X and the form Q.^ 
It is not clear is this approach useful in representation theory of semisimple groups or not. It seems 
that even a problem of existence of nontrivial examples for groups of rank > 1 is open. It seems to me that 
the following question is way to understand this. 

Let G be a semisimple group and K be its compact subgroup. Let an irreducible representation (a 
Harish-Chandra module) p of G in the space X admit a G-invariant Hermitian form Q and let the restriction 
of p to be mutiplicity free. Then X admits a canonical Krein structure (since the restriction of Q to any 
irreducible _ft'-subrepresentation is positive definite or negative definite). The simplest nontrivial examples 
of such a picture are 

1. the representations Ta{g) of the group G = U(p, g) for any noninteger real a. 

2. Molchanov's degenerate representations of 0{p,q) (see p2|). 

There arises the following problem. 

Question A. 4. a) Is it possible in these two cases to write the projections to the subspaces X± 
explicitly? 

b) Are the operators of the representation continuous in the topology of the Krein space? 

A. 8. Indefinite harmonic analysis. Molchanov's approach. This approach is not well- 
formilized, but there is a nice collection of explicit nontrivial examples.^ They have the following form. 

Consider a representation of a group G in a topological vector space X. Let Q be a G-invariant 
Hermitian form on X (the basic example is described above in A. 5). 

Also consider some family pt of irreducible representations of G in the spaces Yt, and assume that each 
representation of this family admits a G-invariant Hermitian forms Rt{-, •)• Consider a space ^ of functions 
/ that takes any t to a. vector yt G It. The space of such functions is some kind of a direct integral of 
representations, but it is not a direct integral in the formal common sense. 

Consider a G-intertwining operator from X to y. The operator J takes any vector a; G A to some 
function ,Jx(t). An indefinite Plancherel formula is the identity 



Q(xi,X2)= J Rt{Jxi(t),Jx2{t)) dfj.{t) 



where dp(t) is some ("Plancherel") measure. 

In particular, this measure is obtained for Berezin forms on certain rank 1 pseudo-Riemannian sym- 
metric spaces (1^ jl|). 

Fix arbitrary (for simplicity noninteger) a G R. Consider the space of smooth functions on Bp., = G/K 
equipped with the (generally indefinite) Hermitian form (4.11). 

Conjecture A. 5. Our Plancherel formula (see Section 7) is valid for arbitrary Q G R in Molchanov's 
sense. 

A. 9. Conformal group of the symmetric space. Each classical pseudo-Riemannian symmetric 
space G/H admits a canonical open embedding to some space G° / P° , where the conformal group^G° D G 
is a classical group and P° D // is some maximal parabolic subgroup in G°^. In fact the space G° / P° is 
a Grassmannian or a product of two Grassmannians. 

^^Let us decribe the simplest example. Consider the space X consisting of finite linear combinations of vectors ei , 
62, . . . ; /l,/2, . ■ . ■ Assume all these vectors are pairwise orthogonal and (cj, ej) = 1 and (/j, fj) = —1. Consider 
the subspace generated by ej and the subspace X— generated by fj . Consider also the subspace X'_^ generated 
by the vectors j -\- lej + sfjfj and the subspace X'_ generated by the vectors \f]ej + \/ j -\- Ifj. Then © X— 
and X'^ (B X'_ are different linear topological spaces containing X, i.e., the identical operator X — > X cannot be 
extended to a bounded bijection Xj^ © X— — > X'_^ © X'_ . 

First example of this kind were observed on "physical level" in |4^ , for mathematically rigorous way of formu- 
lation of such problems see |^^), see also p3| , [isl, 

■^^The term "conformal" was introduced by Goncharov and Gindikin 

^"This fact can be extracted from Makarevich's tables |po|, but it never was claimed before for exceptional 
spaces analogy of this statement, in general, is false 
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Example. In the situation described in 2.4 {G/H = G/K = \]{p,q) /\]{p) x V{q)) the space G° /P° is 
the Grassmannian of p-dimensional subspaces and G° — GL(p + q,C). 

Example. The conformal group in Example 1 in A.l is G° = GL(n, R) x GL(n, K), in Example 2 we 
have GL(2n, C), in Example 3 we have G° = GL(2n, K), in Example 4 we have G° = GL(p + q,C), in 
Example 5 we have G° = Sp(2n,R) x Sp(2n,R), in Example 6 we have G° = Sp(2n,R). 

Remark. The general algorithm for obtaining the conformal group G° is following. 

1. Let a control semiinvolution and a control form be absent. Then G° = G x G. The space G° / P° 
is the product of _B-isotropic Grassmannians if a basic form B is present, and a product of the usual 
Grassmannians otherwise. 

2. Otherwise, we "forget" the control semiinvolution and the control form on V and consider the group 
of automorphisms of the basic form B if the basic form is present, and the complete linear group if B is 
absent. 

Remark. For 44 series of symmetric spaces, the subset G/H is dense in G°/P°. For 10 series G/H 
is not dense in G°/P° but the group G has a finite number of open orbits in G°/P°, all these orbits 
are symmetric spaces of the form G/Hj, and the union of G/Hj is dense in G°/P°. These 10 series are 
distinguished by the condition; a control form is present and it is an Hermitian formf^ 

For Riemannian symmetric spaces G/H (i.e., G/K in the notation of Section 11) the conformal group 
G° and the hidden overgroup G discussed in Section 11 are different objects. Hidden overgroup acts by 
integral operators in and it does not act (even locally) on the symmetric space G/K itself. 

A. 10. Deformation of on pseudo-Riemannian symmetric space. A natural representation 
po of G° in L^{G° /P°) is a representation of a principal degenerate series. 

Observation A.6.([|^) a) IfG has a dense orbit in G°/P°, then the restriction of po to G is L^{G/H). 
Otherwise this restriction is a direct sum of several spaces L^iG/Hj). 

In many cases the representation po can be included in a degenerated complementary series ps. Thus 
there arises a problem of Plancherel formula for the restriction of ps to G. This restriction is some kind of 
deformation of L'^{G/H). 

This problem survives even in the case in which the complementary series is absent. Consider the 
G- invariant kernel on pseudo-Riemannian symmetric space G/H given by 

£x(W^i, ^i; W^2, F2) — X det ■ 



l-D(Wi,yi; 14^2,^2) 



Then the exprssion (A. 2) is an Hermitian form on the space of compactly supported smooth functions on 
G/H. 

Example. For G/H = U(p, g)/U(p) x \J{q) we obtain the U(p, g)-invariant kernel 



|det(l — 0*2) ^ {z* — u*){l — uu*) ^{z — u)^ = 



det(2; — u)(z* — u*) 
det(l — zz") det(l — uu*)y 



This kernel has the main singularity on the diagonal z = u. Evidentely the representation of U(p, g) 
associated with this kernel is not a kernel representation. 
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